Zephyrnet Logo

A Raspberry Pi as a decent residential proxy #RaspberryPi #Networking #PiDay @Raspberry_Pi

Date:

A great use for a Raspberry Pi as a residential proxy server from WiringBits:

One of our projects uses web-scraping to scan several online stores to find discounts. Lately, we started supporting some stores that seem to block requests coming from common cloud providers (like AWS, DigitalOcean, etc), if you are curious, the websites are BestBuy and Costco Mexico.

A popular workaround to mitigate this problem is to pay for a proxy service to scrape these websites, sadly, we weren’t able to find a reliable provider that was within our small budget.

Hence, we ended up building our own residential proxy, right now being powered by an old Raspberry Pi model B. It’s worth adding that it wasn’t as simple as we expected, specially keeping the SSH tunnel available.

The code has been open sourced on GitHub.

I ended up investing more time than expected tweaking the necessary stuff to keep the proxy working reliable, the biggest problem was related the SSH tunnel.

If you see the actual project, it includes a systemd service to keep the tunnel opened with the necessary tweaks.

The tunnel command being:

  • /usr/bin/ssh -nNT -R 9999:localhost:9000 -o ConnectTimeout=10 -o ExitOnForwardFailure=yes -o ServerAliveInterval=180 ubuntu@cazadescuentos.net

What matter the most:

  • ExitOnForwardFailure=yes forces ssh to exit when there is a failure in the connection instead of silently staying running while the tunnel doesn’t work.
  • ServerAliveInterval=180 keeps sending a ping to the server to avoid the server closing the connection due to inactivity.

See the blog post for all the details. Nice work.

Simple proxy flow

Source: https://blog.adafruit.com/2020/06/12/a-raspberry-pi-as-a-decent-residential-proxy-raspberrypi-networking-piday-raspberry_pi/

spot_img

Latest Intelligence

spot_img