HackerRank's Open-Source ATS Exposes How Resume Scoring Really Works — And Why the Scores Are Lying to You

Quick Answer: HackerRank's affiliated repository interviewstreet/hiring-agent is now publicly available MIT-licensed code that scores resumes using an LLM pipeline — revealing exactly how automated resume evaluation works. But the viral story isn't the open source release: it's that the same resume scores 90, then 74, then 88 on consecutive runs, proving ATS scores are non-deterministic noise, not reliable signals.

What Changed: A Black Box Cracked Open

In late 2025, the GitHub repository interviewstreet/hiring-agent — published by InterviewStreet, the entity behind HackerRank — went from a quiet open-source release to a viral LinkedIn and Hacker News sensation. The code is live, MIT-licensed, and fully inspectable. Anyone can clone it, run it locally, and watch their resume get scored in real time.

This matters because for the better part of a decade, ATS (Applicant Tracking System) resume scoring has operated as a black box. Candidates were told to "optimize for ATS" without any visibility into what that actually meant. Career coaches built entire practices around keyword density, formatting rules, and section placement — all of it derived from guesswork, vendor marketing, and reverse-engineering attempts.

Now, for at least one prominent tool, the guesswork ends. The pipeline is readable. The weights are in the code. And what the code reveals is both illuminating and deeply unsettling.

The most viral moment came not from the release itself but from a Hacker News thread where users ran the same resume through the tool repeatedly and received scores of 90, then 74, then 88 — all for identical input. That three-run spread of 16 points is not a rounding error. It is a fundamental characteristic of how LLM-based scoring works, and it quietly invalidates the premise that ATS scores should be used as hiring gatekeepers.

How It Works: Inside the `interviewstreet/hiring-agent` Scoring Pipeline

The tool is a CLI-driven AI agent. Here is what it actually does, step by step, based on the public source code:

Step 1 — PDF Parsing

The agent converts your resume PDF into raw text. This step introduces the first source of variance: PDF-to-text conversion is lossy. Two-column layouts, embedded fonts, tables, and graphics can all produce garbled or missing text depending on the parser's run. A resume that looks perfect visually may produce fragmented text internally.

Step 2 — LLM Extraction (Six Calls)

The agent calls an LLM six separate times to extract structured data from the parsed text:

Basic information (name, contact, summary)
Work history
Education
Technical skills
Projects
Awards and recognitions

Each call is a separate inference. Each inference has some probability of misclassifying, hallucinating, or omitting data — even at low temperature. These six calls compound variability before a single scoring decision is made.

Step 3 — GitHub Enrichment

This is where the tool diverges sharply from traditional ATS. It automatically fetches your GitHub profile, scans your public repositories, classifies each repo by type and quality, and selects your top seven contributions based on commit thresholds. This GitHub signal becomes a significant part of your score.

The implication: two candidates with identical resumes but different GitHub profiles will receive different scores. A candidate with no GitHub account faces a ceiling, regardless of their on-paper experience.

Step 4 — Grading

All extracted data — resume fields plus GitHub context — gets fed into the LLM for a final holistic score. The default local model is gemma3:4b running at temperature 0.1. Production reportedly uses a top-tier Gemini model. The same data, same model, same temperature — and yet scores vary by double digits. This is because LLM inference, even at low temperature, is not perfectly deterministic.

The Scoring Rubric (120 Points Total)

Category	Max Points	What It Measures
Open Source Contributions	35	GitHub activity, community repos, GSoC (+5 bonus)
Personal Projects	30	Complexity, live demos, working GitHub links
Work Experience	25	Internships, full-time roles, startup founder (+5 bonus)
Technical Skills	10	Breadth of languages and frameworks
Bonus Points	+20	GSoC, founding a startup, exceptional signals

The ordering is the editorial statement. Open source contributions — things that most traditional ATS tools do not even check — outweigh work experience by 40%. Technical skills sit at the bottom, worth less than a tenth of the total score. This is almost the inverse of how a conventional keyword-matching ATS operates.

Level up your career with SuperCareer. Daily 10-minute challenges, AI tutoring, and real workplace skills. Try today's challenge free →

Why It Matters for Your Career: Role-by-Role

Software Engineers (actively job-hunting): Your GitHub is now a formal input to scoring, not a "nice to have." A dormant GitHub profile with no public contributions costs you up to 35 points — nearly one-third of the total score — before a human sees your resume. Prioritize pinning your best repositories and ensuring your links are live and functional.

Junior Engineers and New Graduates: The rubric explicitly rewards real-world project complexity and open source over GPA and coursework. Tutorial-only projects on GitHub lose points under this system. A single meaningful contribution to a real open-source library could score higher than two internships listed without GitHub evidence.

Senior Engineers and Tech Leads: The work experience ceiling is 25 points. If you have 12 years of experience but minimal public presence, your ceiling is lower than a junior contributor with an active open source profile. The system does not reward seniority per se — it rewards demonstrability.

Recruiters and Talent Acquisition Leads: The inconsistent scores (90 → 74 → 88 on the same resume) are your credibility problem. If candidates can demonstrate that an ATS tool produced three different scores on a single resume, the argument for using that score as a filter collapses. This is a forcing function to return to skills-based assessment and structured human review.

Career Coaches and Resume Writers: The keyword optimization era may be overstated. If the scoring tool weights open source at 35 points and skills at 10, advising clients to stuff resumes with framework names is misallocating their energy. The new advice: build in public, link everything, keep GitHub active.

Hiring Managers at Startups: This tool explicitly rewards founders with a bonus. If you are screening candidates who have started their own companies, this rubric will surface them. But it also means candidates who have optimized for this specific rubric may game the founder bonus signal with hollow startup claims.

Students and Early-Career Professionals: You have an asymmetric advantage. You have time to build open source contributions that a mid-career professional grinding a day job does not. A well-timed GSoC participation alone earns a 5-point bonus that a 10-year veteran cannot retroactively claim.

Product Managers and Non-Engineers: This specific tool is built around an engineering hiring lens. GitHub enrichment is meaningless if you have no public code repositories. The broader lesson applies: know which ATS your target companies use, and understand that tools optimized for engineers will not score your profile fairly.

How ATS Resume Scoring Works: What the Open Source Reveals That Vendors Never Told You

Before this release, most public understanding of "how ATS resume scoring works" came from three sources: vendor marketing, career coach speculation, and occasional leaks from recruiters. The reality the source code reveals is more complicated — and more honest — than any of those.

Traditional keyword-matching ATS (Workday, Taleo, iCIMS) operate on largely deterministic rule-based logic: does the resume contain a keyword from the job description? The match is literal or uses basic stemming. "Python" matches "Python." "Machine learning" may or may not match "ML" depending on the platform. These systems are rigid but consistent — you get the same score every time.

LLM-based ATS tools like hiring-agent introduce semantic understanding — they can infer that "built distributed systems" implies familiarity with "Kafka" even without the keyword — but they sacrifice consistency for that flexibility. Temperature 0.1 is low, not zero. The model samples from a probability distribution on every token. Run it three times, get three scores.

This distinction is critical for how you approach resume optimization:

For keyword-matching ATS, the optimization is mechanical: match the job description vocabulary, use standard section headers, avoid tables and text boxes that confuse parsers.

For LLM-based ATS, the optimization is qualitative: demonstrate real outcomes, link to live work, provide GitHub evidence, write in plain English that a language model can interpret correctly. Keyword stuffing actively hurts you because it reads as inauthentic.

The parser-vulnerability problem is real and unchanged. Both generations of ATS fail when PDF parsing fails. A beautiful resume in a two-column layout with icons and colored headers may produce garbled text after PDF extraction — and the LLM scores the garbled text, not the visual. Single-column, plain-text-compatible formatting remains the safest foundation regardless of which system you face.

Skills to Learn Now: Your Three-Week Roadmap

Week 1 — Understand the tool and audit your profile

Clone interviewstreet/hiring-agent, install Ollama, pull gemma3:4b, and run your resume through it three times. Compare the scores. The variance you observe is your baseline for how much noise LLM-based scoring introduces. Then read the scoring rubric carefully — you are looking for the delta between where your profile is and where 35 open-source points could take it.

Week 2 — Build demonstrable evidence

Identify one open source project where you can make a meaningful contribution. This does not mean a cosmetic documentation fix. Meaningful contributions have diffs — code, tests, real fixes. Pin your three best repositories on GitHub with accurate README files, live demo links if applicable, and clear descriptions. Remove or unpin tutorial clones and "learning exercise" repos.

Week 3 — Rewrite your resume for humans, not parsers

Use plain single-column formatting. Spell out acronyms on first use. Include concrete outcomes with numbers where possible ("reduced deployment time by 40%", "shipped feature used by 12,000 active users"). Remove keyword stuffing. Link your GitHub profile prominently. Then run the tool again and compare.

Ongoing: Stay current on which ATS your target companies use. Greenhouse, for instance, does not auto-score resumes. Lever uses stemming-based matching. Knowing the system is now possible — do the research.

HackerRank `hiring-agent` vs. Alternatives: The Honest Comparison

Tool	Type	Consistency	GitHub Signal	Transparency	Best For
`interviewstreet/hiring-agent`	LLM-based, open source	Low (±15 pts per run)	Yes (35 pts weight)	Full source code	Engineers who want to audit their scoring
`ats-screener` (sunnypatell)	Rule-based, open source	High	No	Full source code	Simulating Workday/Taleo/iCIMS behavior
Resume Matcher (srbhr)	Semantic NLP, open source	Medium	No	Full source code	Tailoring resume to a specific job description
Jobscan	Proprietary, commercial	High	No	Black box	Matching against a specific job posting
Resume Worded	Proprietary, commercial	High	No	Black box	General ATS readiness with recruiter feedback

The bottom line on alternatives: Rule-based tools like ats-screener are more consistent but miss the semantic richness that modern LLM-based screeners use. Semantic tools like Resume Matcher are better at job-description alignment but do not surface GitHub signals at all. The HackerRank tool's open-source advantage is transparency — you can read exactly why you scored what you scored. Its disadvantage is that the score means less because it changes on repeat runs.

Honest Limitations and Criticism

The score variance problem is not a bug, it is a feature of the architecture — and it's serious. A 16-point swing on the same resume in consecutive runs means the tool cannot be used as a reliable binary filter. If your cutoff score is 75 and a candidate scores 74 on one run and 88 on another, which run governs their candidacy? This is not a theoretical concern — the Hacker News thread documented it empirically.

The rubric is opinionated in ways that favor a specific archetype. A candidate who is an active open source contributor with a populated GitHub, multiple personal projects with live demos, and a startup on their record could score near-perfect — but this describes perhaps 5% of working engineers and almost none of the broader workforce. The rubric implicitly penalizes enterprise engineers who do meaningful closed-source work and cannot publish it.

GitHub enrichment introduces a privacy concern. The tool automatically fetches your public GitHub profile. Candidates without GitHub accounts or with primarily private repositories face a structural disadvantage that has nothing to do with their skill level. This conflates demonstrability with competence.

The tool is optimized for engineers. Non-technical candidates — product managers, designers, operations professionals, data analysts — have no path to the top of this scoring rubric. Running this tool on non-engineering resumes produces scores that are systematically lower and arguably meaningless.

Open-sourcing enables gaming. Now that the rubric is public, candidates who read the code will pin empty or low-quality repositories, claim project complexity they do not have, and fabricate GSoC participation. The tool's designers noted this risk; the mitigating factor is that GitHub enrichment pulls live data, not just self-reported claims. But commit history and repository quality are still manipulable with enough lead time.

The local model (gemma3:4b) and production model (Gemini) are not the same. Running the tool locally to audit your score does not tell you what a company using the production version of the pipeline would see. The rubric may be identical, but the model's interpretation of your data differs between a 4-billion-parameter local model and a frontier Gemini deployment.

Traditional ATS scoring inconsistency is not new. The problem this release makes visible — that ATS scores are unreliable — was always true. Keyword-matching tools reject candidates for missing the word "Python" when they wrote "Py3" throughout. LLM-based tools add semantic richness and subtract consistency. Neither generation is a reliable hiring filter. The open source release just makes the noise quantifiable.

SuperCareer's Take

Learn the system, then build for humans.

The right response to interviewstreet/hiring-agent going viral is not to spend a week tuning your resume for its rubric. It is to understand what this tool reveals about the entire category and invest accordingly.

The rubric's prioritization of open source contributions over technical skills is a signal about where engineering hiring is heading, not just what this tool measures. If public, demonstrable work is becoming the 35-point category and self-reported skills are the 10-point category, the leverage move is to build in public — not to optimize a resume document.

The inconsistency finding (90 → 74 → 88) is equally instructive. If the tool your target company uses produces double-digit score variance, then keyword-optimization is a strategy being applied to a noisy system. The expected return on resume keyword tuning is lower than most candidates assume. The expected return on a well-maintained GitHub, a live side project, and one real open-source contribution is measurably higher — and that work transfers across every company's ATS, not just this one.

For engineers actively job hunting: Clone the repo, run it, understand where you lose points, and invest effort in the categories that matter — public code and demonstrable projects. Do not chase the keyword game.

For recruiters: Use this as a forcing function to supplement ATS scores with structured technical interviews and skills assessments. A tool that produces a 16-point swing on repeat runs should not be the sole filter between a qualified candidate and a conversation.

For career coaches: The rubric is now public. Update your playbook. The 2020s advice to stuff resumes with keywords is increasingly misaligned with how modern scoring actually works.

The experiment is: read the source code yourself. It takes 20 minutes. Most of your competition will not bother. The ones who do will understand the system more accurately than any career advice article — including this one — can convey in prose.

Frequently Asked Questions

Why does my ATS resume score change every time I submit?

LLM-based ATS tools, including the interviewstreet/hiring-agent, use language models that are probabilistic even at low temperature settings. Each inference samples from a probability distribution, so identical inputs produce slightly different outputs across runs. This is inherent to the architecture, not a bug — but it makes the scores unreliable as hard cutoff filters.

How does HackerRank's open-source tool score resumes?

The interviewstreet/hiring-agent pipeline parses your resume PDF into text, calls an LLM six times to extract structured data (work history, skills, education, projects, awards), enriches the profile with live GitHub data, and produces a score out of 120 points weighted heavily toward open source contributions (35 pts), personal projects (30 pts), and work experience (25 pts).

Can I game an open-source ATS if I can read the code?

To a degree, yes. Knowing the rubric lets you optimize directly — pin quality GitHub repositories, ensure project links are live, highlight open source contributions. However, the GitHub enrichment component pulls live data and evaluates actual commit history and repository complexity, which limits how much you can fabricate without doing real work.

What do ATS systems actually look for in a resume?

It depends on the generation of the tool. Traditional keyword-matching ATS (Workday, Taleo) look for exact or stemmed keyword matches between the resume and job description. LLM-based tools like interviewstreet/hiring-agent evaluate semantic content, project quality, GitHub activity, and career arc holistically. The scoring dimensions differ significantly between generations.

Is ATS resume scoring reliable for hiring decisions?

For LLM-based tools, the evidence suggests no. A 16-point variance across three runs on the same resume means the score cannot function as a reliable binary filter. Traditional keyword-matching tools are more consistent but miss qualified candidates who use different vocabulary. Both generations have known failure modes that should prevent exclusive reliance on automated scores.

Does keyword stuffing still work on modern ATS tools?

For legacy keyword-matching systems (Taleo, many Workday configurations), keyword presence still matters. For LLM-based systems, keyword stuffing is counterproductive — the model evaluates overall quality and coherence, and a resume padded with disconnected keywords reads as low-quality. Match the vocabulary of the job description naturally; do not repeat keywords mechanically.

Should recruiters trust ATS scores to filter candidates?

No tool should be the sole filter. ATS scores — whether rule-based or LLM-based — surface patterns and reduce initial volume, but each generation has documented failure modes. The open-source release of interviewstreet/hiring-agent makes those failure modes visible and quantifiable for the first time. Use scores as one signal among several, not as a gatekeeping cutoff.

How do I optimize my resume for automated screening?

Use a single-column, plain-text-compatible format so PDF parsing produces clean output. Match key terms from the job description naturally in context. Ensure all project links and GitHub URLs are live. For LLM-based tools, write in clear, outcome-oriented language that a language model can interpret correctly. For traditional ATS, prioritize keyword match over prose quality in the skills section.

Join the SuperCareer AI career newsletter for your personalized roadmap — weekly intel on the tools, skills, and moves that actually move your career forward.