Every job in your Zorlya feed came from one of 130+ sources. The sources are listed in the app after you sign in, and each one has a rule in common: it allows a program to read it. A board that forbids that is not in the list, however many jobs it carries. This post explains which sources made it, how they differ, and why some large names are missing.
Two kinds of source
Public job boards. These are sites that collect postings from many employers and publish them as a feed or an open API. We Work Remotely, RemoteOK, Remotive, Himalayas and the monthly Hacker News “Who is hiring” thread are here. So are narrower boards for one language or community: WordPress.org Jobs, LaraJobs for Laravel, Python.org Jobs, GolangProjects and Elixir Jobs. Some are national public boards, such as Job Bank in Canada and MyCareersFuture in Singapore. Hasjob, a startup board that is mostly India, is read through its public Atom feed.
Applicant tracking systems. An applicant tracking system, or ATS, is the software a company uses to post its openings and collect applications. Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Workday, BambooHR and others are in the list. Reading these means reading the employer’s own board, so the posting is the original, with the full ad text, and not a copy that a board made and may have shortened.
The ATS sources are read company by company. A board on Greenhouse belongs to one employer, so Zorlya keeps a list of companies to read on each platform. When postings on other boards link to a company board we do not read yet, an administrator can see that company in a list and add it, and its next openings then come straight from the source.
A few more sources are built and registered but are not in that count, because they read nothing until a key or a login is set. Their jobs are not in your feed.
Reading politely
Some sources need more than one request per job. Rippling, SmartRecruiters, BambooHR and Workday list openings without their descriptions, so each posting kept costs a second call. Each of them has a budget of 300 such calls a run, spread across its companies in turn, so one large employer cannot use the whole budget.
Job Bank asks for 5 seconds between requests in its robots.txt, so it gets one request a run, with no retry.
Where an employer has chosen to hide its name, Zorlya keeps it hidden. MyCareersFuture lets an employer post through an agency without naming itself, and the field that would reveal the employer is never read. Salaries marked private stay private too.
The ones we turned down
More than 20 sites were checked and left out. Each one was measured or read before the decision, and the reason is recorded so nobody has to check again.
LinkedIn, Indeed and Glassdoor. None of the three publishes a public API, and all three forbid scraping in their terms. Most employers who post there also post on an ATS we read, so their jobs still arrive, from the source.
Boards that refuse readers. bdjobs in Bangladesh and Dice in the US disallow automated reading in their robots.txt, and Wellfound’s terms forbid it. The robots.txt of bdjobs ends with a rule that turns away every reader it has not named. We do not change our crawler’s name to get through a door addressed to someone else.
Boards with too little in them. Jobspresso’s feed held no job items when it was checked. A source that adds almost nothing is not worth the requests it costs the site.
When a country’s boards all refuse
Bangladesh is the clearest case. Every large board there refuses automated reading.
The way in turned out to be employers, not boards. Several large Bangladeshi employers post on platforms we already read: ShopUp and Sheba on SmartRecruiters, and Grameenphone on Manatal’s Careers Page. Adding them costs one line in the company list, not a new source. The same approach works anywhere the boards are closed and the employers are not. Check where a company posts before concluding a country cannot be reached.
How does a job get from a source to your feed?
Every source follows the same path. The steps are the same whether the source is a national board or one company’s ATS.
- The source is due. Each source has an interval, learned from how many new jobs it usually brings, between 1 and 6 hours by default. An administrator can set a different one for a source.
- The reader fetches the list. For most boards it reads the site’s robots.txt first and follows it, and it keeps to the request budget set for that source.
- The fields are kept. The title, the employer, the date and the link, and the place when the source gives one. The description, salary and other details only where the source provides them and its terms allow Zorlya to keep them.
- Duplicates are matched. A posting at the same address as one already held, or with the same title, company, place and text, shows as one job.
- The job is scored. Each new job is scored against the profiles that target its kind of work, and it appears in their feeds.
What is robots.txt, and why does it decide?
robots.txt is a small file at the root of a website that tells automated readers which parts of the site they may visit. The standard that defines it, RFC 9309, says plainly that the rules are not a form of access authorisation: a reader that ignores them can still get in. That is why following them is a choice. Zorlya reads a site only where its robots.txt and its terms allow it, and when they do not, the site is left out, as the sections above show.
What does Zorlya keep from a posting?
It depends on what the source licenses. Where a site’s terms do not allow its postings to be republished, Zorlya keeps only five things: the title, the employer, the place, the date and the link. You then read the full ad on the site itself. Where a source allows more, such as an employer’s own ATS board, the full ad text is kept, and the score has more to work from.
Is a job in my feed still open?
Not always, and this is a limit worth knowing. Zorlya does not close a posting when a board stops listing it, so a role can stay in your feed after the employer has filled it. Before you spend time on an application, open the original posting and check that it is still there and still taking applications. The feed’s posted date filter helps you keep to recent roles. Why a job’s posted date is unreliable explains why the date itself needs care.
What this means for your feed
You will not see every job on the internet. You will see jobs from sources that allow it, read from the original where we can, with the employer’s own description. When a board and the employer’s own ATS carry the same posting at the same address, or with the same title, company, place and text, you see one job. A copy that differs in any of those stays a separate job.
The source of every posting is shown on it, so you can always see where a job came from and open the posting it was read from. The app lists every one after you sign in.
Questions people ask
Why is a job I saw elsewhere not in Zorlya?
Either its site does not allow automated reading, or the employer posts on a board Zorlya does not read. In the app you can add it yourself from its link or its text, and it stays private to your account.
Does Zorlya scrape sites?
Not against a site’s rules. It reads feeds, public APIs and, where robots.txt and the terms allow it, the structured data on job pages. A site that refuses automated reading is left out.
How often are new jobs added?
Each source is read on its own interval, usually between 1 and 6 hours, so new jobs arrive through the day.
Can I ask for a source or an employer to be added?
Yes. The employer directory links to a page where you can suggest a company, and the team reviews each suggestion.
What we know and what we do not
Every source Zorlya reads is listed in the app after you sign in, with the kind of source it is. A source that needs a key reads nothing until the key is set. We do not know how many jobs exist that no permitted source carries, and we do not close a posting that has vanished from its board, so the original page is the one to trust for whether a role is still open. You can suggest an employer that is missing.