THE X ALGORITHM PLAYBOOK: How X's New Rust+Python Architecture Changes Everything

Follow @nibzard for more insights on algorithms and growth strategies.

THE X ALGORITHM PLAYBOOK: How X’s New Rust+Python Architecture Changes Everything §

Table of Contents §

1. Executive Summary: X’s New Architecture
2. Thunder: In-Network Post Store
3. Phoenix Retrieval: Two-Tower Model
4. Phoenix Ranking: Grok Transformer
5. Home Mixer: Orchestration Pipeline
6. Candidate Pipeline Framework
7. Multi-Action Prediction
8. Content Strategy for New Algorithm
9. Key Differences from Old Algorithm

Executive Summary: Understanding Twitter’s Recommendation Engine §

X’s recommendation algorithm underwent a complete rewrite, transitioning from Scala to a Rust+Python hybrid architecture. This new system fundamentally changes how content is retrieved, ranked, and served to users.

Core Architecture Changes §

The old pipeline architecture has been replaced with a microservices-based system:

Thunder (In-Network Store) → Phoenix Retrieval (Two-Tower) → Phoenix Ranking (Grok) → Home Mixer (Rust/gRPC)

Key Architectural Shifts §

Performance Characteristics §

Thunder provides sub-millisecond post lookups from followed accounts via real-time Kafka ingestion.

Phoenix Retrieval reduces millions of candidates to thousands using two-tower embedding similarity.

Phoenix Ranking uses a Grok-based transformer with candidate isolation — candidates don’t attend to each other.

Home Mixer orchestrates the entire pipeline in Rust with strict resource controls.

Thunder: In-Network Post Store §

Thunder is X’s real-time, in-network post store built in Rust. It replaces the complex candidate source system with a straightforward, high-performance cache.

Architecture §

// From thunder_service.rs
pub struct ThunderServiceImpl {
    /// PostStore for retrieving posts by user ID
    post_store: Arc<PostStore>,
    /// StratoClient for fetching following lists when not provided
    strato_client: Arc<StratoClient>,
    /// Semaphore to limit concurrent requests and prevent overload
    request_semaphore: Arc<Semaphore>,
}

Key Features §

1. Real-Time Kafka Ingestion
- Posts arrive via Kafka streams
- Automatically stored in memory
- Retention trimming handled automatically

2. Sub-Millisecond Lookups

// Fetch all posts (original + secondary) for the followed users
let all_posts: Vec<LightPost> = post_store.get_all_posts_by_users(
    &following_user_ids,
    &exclude_tweet_ids,
    start_time,
    request_user_id,
);

3. Request Throttling

// Try to acquire semaphore permit without blocking
let _permit = match self.request_semaphore.try_acquire() {
    Ok(permit) => permit,
    Err(_) => return Err(Status::resource_exhausted("Server at capacity, please retry")),
};

4. Scoring by Recency

fn score_recent(mut light_posts: Vec<LightPost>, max_results: usize) -> Vec<LightPost> {
    light_posts.sort_unstable_by_key(|post| Reverse(post.created_at));
    light_posts.into_iter().take(max_results).collect()
}

What This Means for Content §

Freshness is everything: Thunder only serves recent content from followed accounts. The recency scoring means the newest posts get priority.

In-network advantage: Posts from accounts you follow are retrieved directly from Thunder’s cache, giving them priority over out-of-network content.

Video optimization: Thunder has specialized video retrieval:

if req.is_video_request {
    post_store.get_videos_by_users(&following_user_ids, ...)
}

Phoenix Retrieval: Two-Tower Model §

Phoenix Retrieval implements a two-tower neural architecture for efficient candidate retrieval — a fundamental shift from the hand-crafted SimClusters system.

Architecture §

# From recsys_retrieval_model.py
class PhoenixRetrievalModel(hk.Module):
    """A two-tower retrieval model using the Phoenix transformer for user encoding.

    - User Tower: Encodes user features + history using the Phoenix transformer
    - Candidate Tower: Projects candidate embeddings to a shared space
    """

User Tower: Encoding User State §

def build_user_representation(
    self,
    batch: RecsysBatch,
    recsys_embeddings: RecsysEmbeddings,
) -> Tuple[jax.Array, jax.Array]:
    """Build user representation from user features and history.

    Returns:
        user_representation: L2-normalized user embedding [B, D]
        user_norm: Pre-normalization L2 norm [B, 1]
    """

The user tower combines:
- User hashes → User embeddings (multi-hash projection)
- History post hashes → Post embeddings
- History author hashes → Author embeddings
- Action embeddings → Multi-hot action vectors
- Product surface embeddings → App/platform context

These are encoded by the Phoenix transformer and pooled to create a single user representation vector.

Candidate Tower: Projecting Content §

class CandidateTower(hk.Module):
    """Candidate tower that projects post+author embeddings to a shared embedding space.
    """

    def __call__(self, post_author_embedding: jax.Array) -> jax.Array:
        proj_1 = hk.get_parameter("candidate_tower_projection_1", [...])
        proj_2 = hk.get_parameter("candidate_tower_projection_2", [...])

        hidden = jnp.dot(post_author_embedding, proj_1)
        hidden = jax.nn.silu(hidden)
        candidate_embeddings = jnp.dot(hidden, proj_2)

        # L2 normalize for efficient dot-product similarity
        candidate_representation = candidate_embeddings / jnp.sqrt(jnp.sum(candidate_embeddings**2))

Retrieval via Dot Product §

def _retrieve_top_k(
    self,
    user_representation: jax.Array,
    corpus_embeddings: jax.Array,
    top_k: int,
) -> Tuple[jax.Array, jax.Array]:
    """Retrieve top-k candidates from a corpus for each user.

    Uses dot product similarity (valid due to L2 normalization).
    """
    scores = jnp.matmul(user_representation, corpus_embeddings.T)
    top_k_scores, top_k_indices = jax.lax.top_k(scores, top_k)

What This Means for Content §

Learned embeddings: Your content is represented by learned embeddings rather than hand-crafted features. The model learns what makes content similar.

Author identity matters: The candidate tower explicitly incorporates author embeddings alongside post embeddings.

Multi-action context: The user tower considers all types of engagement (likes, replies, retweets, profile clicks, etc.) when building the user representation.

Phoenix Ranking: Grok Transformer §

Phoenix Ranking uses a Grok-based transformer for final scoring — a unified model replacing the separate Light/Heavy ranker architecture.

Architecture §

class PhoenixModel(hk.Module):
    """A transformer-based recommendation model for ranking candidates.
    """

    model: Transformer  # Grok transformer architecture
    config: PhoenixModelConfig

Input Construction §

def build_inputs(
    self,
    batch: RecsysBatch,
    recsys_embeddings: RecsysEmbeddings,
) -> Tuple[jax.Array, jax.Array, int]:
    """Build input embeddings from batch and pre-looked-up embeddings.

    Returns:
        embeddings: [B, 1 + history_len + num_candidates, D]
        padding_mask: [B, 1 + history_len + num_candidates]
        candidate_start_offset: int - position where candidates start
    """

The input sequence is structured as:

[USER_EMBEDDING, HISTORY_TOKENS..., CANDIDATE_1, CANDIDATE_2, ...]

Candidate Isolation §

The key innovation is candidate isolation — candidates don’t attend to each other:

model_output = self.model(
    embeddings,
    padding_mask,
    candidate_start_offset=candidate_start_offset,  # Prevents cross-candidate attention
)

This means each candidate is scored independently based on the user and history, without influence from other candidates in the batch.

Multi-Action Prediction §

def __call__(self, batch: RecsysBatch, recsys_embeddings: RecsysEmbeddings) -> RecsysModelOutput:
    """Forward pass for ranking candidates.

    Returns:
        RecsysModelOutput containing logits for each candidate.
        Shape = [B, num_candidates, num_actions]
    """

The model predicts probabilities for all engagement types simultaneously:
- Like
- Reply
- Retweet
- Quote
- Profile click
- And 10+ more action types

Final Scoring §

unembeddings = self._get_unembedding()  # [emb_size, num_actions]
logits = jnp.dot(candidate_embeddings, unembeddings)

What This Means for Content §

Unified scoring: No more separate Light/Heavy ranker — the Grok transformer handles everything.

Candidate independence: Your content is scored on its own merits, not relative to other candidates.

Multi-action optimization: The model considers all possible engagement types, so content that drives diverse engagement is favored.

Home Mixer: Orchestration Pipeline §

Home Mixer is the Rust-based orchestration layer that coordinates the entire recommendation pipeline via gRPC.

Architecture §

// From server.rs
pub struct HomeMixerServer {
    phx_candidate_pipeline: Arc<PhoenixCandidatePipeline>,
}

impl pb::scored_posts_service_server::ScoredPostsService for HomeMixerServer {
    async fn get_scored_posts(
        &self,
        request: Request<pb::ScoredPostsQuery>,
    ) -> Result<Response<ScoredPostsResponse>, Status> {
        let pipeline_result = self.phx_candidate_pipeline.execute(query).await;
        // ...
    }
}

Pipeline Execution §

// From candidate_pipeline.rs
async fn execute(&self, query: Q) -> PipelineResult<Q, C> {
    let hydrated_query = self.hydrate_query(query).await;
    let candidates = self.fetch_candidates(&hydrated_query).await;
    let hydrated_candidates = self.hydrate(&hydrated_query, candidates).await;
    let (kept_candidates, _) = self.filter(&hydrated_query, hydrated_candidates).await;
    let scored_candidates = self.score(&hydrated_query, kept_candidates).await;
    let selected_candidates = self.select(&hydrated_query, scored_candidates);
    // ...
}

Pipeline Stages §

1. Query Hydrators — Enrich the request with user context

async fn hydrate_query(&self, query: Q) -> Q {
    let hydrators: Vec<_> = self.query_hydrators().iter().filter(|h| h.enable(&query)).collect();
    // Run in parallel and merge results
}

2. Sources — Fetch candidates from multiple sources

async fn fetch_candidates(&self, query: &Q) -> Vec<C> {
    let sources: Vec<_> = self.sources().iter().filter(|s| s.enable(query)).collect();
    // Thunder, Phoenix, etc. run in parallel
}

3. Hydrators — Enrich candidates with features

async fn hydrate(&self, query: &Q, candidates: Vec<C>) -> Vec<C> {
    // Add author info, media info, etc.
}

4. Filters — Remove low-quality candidates

async fn filter(&self, query: &Q, candidates: Vec<C>) -> (Vec<C>, Vec<C>) {
    // Duplicate removal, age filters, muted authors, etc.
}

5. Scorers — Apply ML models

async fn score(&self, query: &Q, candidates: Vec<C>) -> Vec<C> {
    // Phoenix ranking, weighted multi-action scores
}

6. Selectors — Choose top-K

fn select(&self, query: &Q, candidates: Vec<C>) -> Vec<C> {
    // Sort by score and return top-K
}

7. Side Effects — Async operations (caching, logging)

fn run_side_effects(&self, input: Arc<SideEffectInput<Q, C>>) {
    tokio::spawn(async move {
        // Non-blocking operations
    });
}

What This Means for Content §

Parallel execution: Multiple sources are queried simultaneously — content from Thunder (in-network) and Phoenix (out-of-network) competes on equal footing.

Composable architecture: The pipeline is modular — new sources, filters, and scorers can be added without restructuring.

Strict resource controls: The pipeline enforces limits at each stage to ensure consistent performance.

Candidate Pipeline Framework §

The new algorithm uses a composable trait-based framework for building recommendation pipelines.

Trait-Based Architecture §

#[async_trait]
pub trait CandidatePipeline<Q, C>: Send + Sync {
    fn query_hydrators(&self) -> &[Box<dyn QueryHydrator<Q>>];
    fn sources(&self) -> &[Box<dyn Source<Q, C>>];
    fn hydrators(&self) -> &[Box<dyn Hydrator<Q, C>>];
    fn filters(&self) -> &[Box<dyn Filter<Q, C>>];
    fn scorers(&self) -> &[Box<dyn Scorer<Q, C>>];
    fn selector(&self) -> &dyn Selector<Q, C>;
}

Pipeline Stage Traits §

QueryHydrator — Enrich the query before candidate fetching

pub trait QueryHydrator<Q>: Send + Sync {
    fn enable(&self, query: &Q) -> bool;
    fn hydrate(&self, query: &Q) -> impl Future<Output = Result<Hydrated>>;
}

Source — Fetch candidates from a data source

pub trait Source<Q, C>: Send + Sync {
    fn enable(&self, query: &Q) -> bool;
    fn get_candidates(&self, query: &Q) -> impl Future<Output = Result<Vec<C>>>;
}

Hydrator — Add features to candidates

pub trait Hydrator<Q, C>: Send + Sync {
    fn enable(&self, query: &Q) -> bool;
    fn hydrate(&self, query: &Q, candidates: &[C]) -> impl Future<Output = Result<Vec<Hydrated>>>;
}

Filter — Remove candidates that don’t meet criteria

pub trait Filter<Q, C>: Send + Sync {
    fn enable(&self, query: &Q) -> bool;
    fn filter(&self, query: &Q, candidates: Vec<C>) -> impl Future<Output = Result<FilterResult<C>>>;
}

Scorer — Apply scoring models

pub trait Scorer<Q, C>: Send + Sync {
    fn enable(&self, query: &Q) -> bool;
    fn score(&self, query: &Q, candidates: &[C]) -> impl Future<Output = Result<Vec<Scored>>>;
}

Selector — Choose top candidates

pub trait Selector<Q, C>: Send + Sync {
    fn enable(&self, query: &Q) -> bool;
    fn select(&self, query: &Q, candidates: Vec<C>) -> Vec<C>;
}

Error Handling §

for source in sources.iter() {
    match source.get_candidates(query).await {
        Ok(mut candidates) => collected.append(&mut candidates),
        Err(err) => error!("source={} failed: {}", source.name(), err),
    }
}

Each stage continues even if individual components fail — failures are logged but don’t halt the pipeline.

What This Means for Content §

Multiple sources: Content can come from Thunder (in-network), Phoenix (neural retrieval), or other sources — all compete equally.

Graceful degradation: If one source fails, others continue — content has multiple pathways to reach users.

Extensibility: New recommendation strategies can be added by implementing the traits — the algorithm can evolve without restructuring.

Multi-Action Prediction §

The new algorithm predicts multiple engagement types simultaneously, a significant shift from the single-score approach of the old algorithm.

Action Embedding Projection §

def _get_action_embeddings(self, actions: jax.Array) -> jax.Array:
    """Convert multi-hot action vectors to embeddings.

    Uses a learned projection matrix to map the signed action vector
    to the embedding dimension. This works for any number of actions.
    """
    action_projection = hk.get_parameter("action_projection", [num_actions, D])
    actions_signed = (2 * actions - 1).astype(jnp.float32)  # Convert to ±1
    action_emb = jnp.dot(actions_signed, action_projection)

History actions are converted from multi-hot vectors (e.g., [1, 0, 1, 0, ...]) to continuous embeddings.

Multi-Action Output §

class RecsysModelOutput(NamedTuple):
    """Output of the recommendation model."""
    logits: jax.Array  # Shape: [B, num_candidates, num_actions]

The model outputs logits for all action types for each candidate:
- Like
- Reply
- Retweet
- Quote
- Profile click
- Video quality view (10s or 95% completion)
- Video 50% playback
- And more…

Final Score Computation §

The final score is typically a weighted sum of action probabilities:

final_score = sum(action_probability[i] * weight[i] for i in all_actions)

Weights differ by product surface (For You vs. Following) and user preferences.

What This Means for Content §

Diverse engagement wins: Content that drives multiple engagement types is favored over content that drives only one.

Action-specific optimization: You can optimize for specific actions (e.g., profile clicks) rather than a generic “score.”

Context matters: The same action may have different weights depending on the user and surface.

Content Strategy: Algorithm-Friendly Formats §

The architectural changes significantly impact effective content strategies.

In-Network Advantage (Thunder) §

Prioritize followed accounts: Thunder gives in-network content a natural advantage through direct cache access.

Post consistently: Thunder’s recency scoring means fresh posts from followed accounts are prioritized.

Video focus: Thunder has specialized video retrieval paths.

Embedding Optimization (Two-Tower) §

Author consistency: The candidate tower incorporates author embeddings — consistent style and topic helps.

Engagement diversity: The user tower considers all engagement types — varied engagement improves user embeddings.

Niche alignment: Learned embeddings cluster similar content — topical consistency helps retrieval.

Candidate Isolation (Grok) §

Standalone value: Content is scored independently — each post should provide value on its own.

User-history alignment: The model attends to user history — content should align with your audience’s interests.

Multi-action design: Design for multiple engagement types (comments, shares, saves).

Pipeline Awareness §

Pass filters: Duplicate, age, and quality filters remove candidates — avoid spam triggers.

Score high enough: Content must survive filtering to reach ranking — maintain baseline quality.

Top-K selection: Only top candidates are served — competitive quality matters.

Key Differences from Old Algorithm §

Language and Architecture §

Retrieval §

Ranking §

Candidate Sources §

What Stays the Same §

• Freshness matters: Content age is still critical
• Engagement quality: Quality signals still trump raw volume
• Social proof: Network effects still amplify reach
• Spam filtering: Quality standards are enforced

What Changes §

• Embedding-driven: Hand-crafted features → learned representations
• Unified ranking: No more Light/Heavy distinction
• Microservices: Modular, composable architecture
• Multi-action: Explicit multi-action prediction

Implementation Guide: Step-by-Step Growth Plan §

Phase 1: Embedding Alignment (Weeks 1-4) §

Understand the two-tower model:
1. Consistent topical focus — Helps the candidate tower learn your content embedding
2. Author brand consistency — Author embeddings are part of the representation
3. Engagement diversity — User towers consider all engagement types

Phase 2: Thunder Optimization (Weeks 5-8) §

Leverage in-network advantages:
1. Follow-for-follow growth — More followers = more Thunder reach
2. Consistent posting — Recency scoring favors fresh content
3. Video focus — Specialized Thunder video paths

Phase 3: Multi-Action Strategy (Weeks 9-12) §

Design for diverse engagement:
1. Conversation starters — Drive replies
2. Shareable insights — Drive retweets
3. Value propositions — Drive profile clicks
4. Video content — Drive quality views

Phase 4: Pipeline Compliance (Weeks 13-16) §

Ensure passage through all stages:
1. Pass filters — Avoid duplicates, spam triggers
2. Score well — Multi-action optimization
3. Top-K selection — Competitive quality

Conclusion: Sustainable Growth on Twitter §

X’s new algorithm represents a fundamental architectural shift from Scala to Rust+Python, from hand-crafted features to learned embeddings, from monolithic to microservices.

Key Takeaways §

1. Two-tower retrieval — Learned embeddings replace hand-crafted features
2. Grok transformer — Unified ranking replaces Light/Heavy split
3. Thunder in-network — Real-time post store for followed accounts
4. Candidate isolation — Content scored independently
5. Multi-action prediction — All engagement types modeled simultaneously
6. Composable pipeline — Trait-based framework enables evolution

Strategic Implications §

For creators:
- Embedding consistency matters more than individual optimization
- In-network growth provides direct Thunder access
- Multi-action content outperforms single-focus posts

For businesses:
- Author brand consistency helps candidate tower learning
- Diverse engagement improves user tower representations
- Pipeline compliance is non-negotiable

For developers:
- Microservices architecture enables A/B testing
- Composable traits allow rapid experimentation
- Rust performance + Python ML flexibility

The new algorithm is more modular, more neural, and more adaptable. While the core principles of quality, engagement, and freshness remain, the mechanisms have fundamentally changed.

About This Analysis: This playbook is based on analysis of X’s open-sourced algorithm code (Rust + Python), including Thunder, Phoenix retrieval/ranking, Home Mixer, and the candidate pipeline framework.

Disclaimer: X’s algorithm is constantly evolving. While this analysis is based on the current open-sourced code, the platform may update its algorithms at any time.

Follow Me for More

For more insights on algorithms, growth strategies, and tech analysis, follow me on Twitter: @nibzard