AI-enhanced for better readability
Profitmind | Web Scraping Junior Developer
Location: Remote or Pittsburgh Salary: $90,000 - $110,000, Full-time Source: hackernews Link: https://www.profitmind.com/
About the Role
At Profitmind, we're building massive-scale ecommerce datasets to use in AI training/inference. We need a junior engineer to help develop the scraping infrastructure for product data.
Responsibilities
- Reverse-engineer undocumented APIs
- Handle anti-bot systems
- Deal with edge cases like pagination limits, rate limiting, and sites that change their protection schemes without warning
- Analyze a site's network requests
- Deobfuscate and read obfuscated javascript
- Implement simple HTTP request scraping to full browser automation
- Work on the infrastructure layer: state management for resumable scrapes, deduplicating products, data integrity, and monitoring systems to detect when sites change
Requirements
- Solid Python skills
- Experience scraping ecommerce websites and APIs
- Enjoy the slightly-obsessive investigative nature of the work
Contact
Reach out directly - gray at netail.ai
If you worked in scraping or botting in the past, please hit me up!