Connect with us

Publications

How POST Requests with Python Make Web Scraping Easier

Published

on

image

Otávio Simões Hacker Noon profile picture

@otaviossOtávio Simões

Economist and data enthusiast

When scraping a website with Python, it’s common to use the

urllib

or the

Requests

libraries to send

GET

requests to the server in order to receive its information. 

However, you’ll eventually need to send some information to the website yourself before receiving the data you want, maybe because it’s necessary to perform a log-in or to interact somehow with the page.

To execute such interactions, Selenium is a frequently used tool. However, it also comes with some downsides as it’s a bit slow and can also be quite unstable sometimes. The alternative is to send a

POST

request containing the information the website needs using the request library.

In fact, when compared to Requests, Selenium becomes a very slow approach since it does the entire work of actually opening your browser to navigate through the websites you’ll collect data from. Of course, depending on the problem, you’ll eventually need to use it, but for some other situations, a

POST

request may be your best option, which makes it an important tool for your web scraping toolbox.

In this article, we’ll see a brief introduction to the

POST

method and how it can be implemented to improve your web scraping routines.

Web Scraping 

Although

POST

requests are commonly used to interact with APIs, they are also useful to fill HTML forms in a website or perform other actions automatically. 

Being able to perform such tasks is an important ability while web scraping, as it’s not rare to have to interact with the web page before reaching the data, you’re aiming to scrape.

Identifying an HTML Form

Before you start sending information to the website, you first need to understand how it will receive such information. Let’s say the idea is to log in to your account. If the site receives the username and password through an HTML form, it will probably look like this:

If that’s the case, all you have to do is send the username and the password within your

POST

request.

But how to identify and even see what the HTML form looks like? For this, we can go back to our old friend: the

GET

request. With a

GET

and using BeautifulSoup to parse the HTML, it’s easy to see all the HTML forms on the page and how each of them looks like.

This is a simple code for this task:

import requests
from bs4 import BeautifulSoup
page = requests.get('http://website.com').text
soup = BeautifulSoup(page, 'html.parser')
forms = soup.find_all('form')
for form in forms: print(form)

And this is how our simple login form that will be the output of the code above:

<form action="login.html" method="post"> User Name: <input name="username" type="text"/><br/> Password: <input name="password" type="text"/><br/> <input id="submit" type="submit" value="Submit"/>
</form>

In a form like this, the “action” is where in the website you should send your request, and the “username” and “password” are the fields you want to fill. You can also notice the type for these values is specified as text.

Submitting Your First POST 

Now it’s time to send your first

POST

request. A basic request will contain only two arguments: the URL that will receive the request and the data that you’re sending.

The data is usually a dictionary where the keys are the names of the fields you intend to fill, and values are what you’re going to fill the fields with. The data can also be passed in different ways, but that’s a more complex approach that’s out of scope for this article.

The code is pretty simple. Actually, you can get it done with only two lines of code:

payload = {'username': 'user', 'password': '1234'}
r = requests.post('http://website.com/login.html', data=payload)
print(r.status_code)

The third line of code is just so you can see the status code of your request. You want to see a status code of 200, which means everything is OK. To learn more about it, click here.

We can now make this process more sophisticated by implementing the

POST

request we just created into a function. Here’s how it’ll work:

1. The

post_request

function will receive two arguments: the URL and the payload to send the request.

2. Inside the function, we’ll use a

try

and an

except

clause to have our code ready to handle a possible error.

3. If the code doesn’t crash and we receive a response from the server, we’ll then check if this response is the one we’re expecting. If so, the function will return it.

4. If we get a different status code, nothing will be returned, and the status will be printed.

5. If the code raises an exception, we’ll want to see what happened, and so the function will print this exception.

And this is the code for all this:

def post_request(url, payload): try: r = requests.post(url, data=payload) if r.status_code == 200: return r else: print(r.status_code) except Exception as e: print(e)

Depending on the website, however, you’ll need to deal with other issues in order to actually perform a login. The good news is that the Requests library provides resources to deal with cookies, HTTP authentications, and more that will have you covered. The goal here was just to use a common type of form as an easy example to understand for someone that had never used a

POST

request before.

Final Considerations

Especially if you’re sending a lot of requests to a particular website, you might want to insert some random pauses in your code in order not to overload the server and use even more

try

and

except 

clauses throughout your code and not only in the

post_request

function to make sure it’s prepared to handle other exceptions it may find along the way. 

Of course, it’s also a good practice to take advantage of a proxy provider, such as Infatica, to make sure your code will keep running as long as there are requests left to submit and data to be collected, and that you and your connection are protected.

The idea of this article is to be only an introduction to

POST 

requests and how they can be useful for collecting data on the web. We basically went through how to fill out a form automatically and even how to log in to a website, but there are also other possibilities such as marking a check box or selecting items from a dropdown list, for instance, which could be subject to an entirely new article.

I hope you’ve enjoyed this and that it can maybe be useful somehow. If you have a question, a suggestion, or just want to be in touch, feel free to be in touch.

Tags

Join Hacker Noon

Create your free account to unlock your custom reading experience.

Coinsmart. Beste Bitcoin-Börse in Europa
Source: https://hackernoon.com/how-post-requests-with-python-make-web-scraping-easier-9i203511?source=rss

CNBC

Louis Vuitton is making a mobile game with embedded NFTs

Published

on

The worlds of fashion and gaming are cozying up to one another. From Gucci selling digital items in Roblox to Vogue‘s virtual fashion spread starring Gigi Hadid, the two industries are increasingly overlapping to capture a young, tech-savvy audience. Now, Louis Vuitton is jumping on the bandwagon by releasing a mobile video game to celebrate its founder’s 200th birthday. Louis: The Game stars the fashion house’s mascot Vivienne, a wooden doll embellished with the company’s flower insignia, on a journey through the brand’s history, reports Austrian newspaper Kurier.

The iOS and Android game reportedly tasks players with exploring a colorful world by completing various quests — all pretty generic so far. Whether it turns out to be more than just an unashamed bit of promotional fluff remains to be seen. The game drops on August 4th. If LV goes all-out psychedelic like it did with its recent UFO-style speaker, then it could be worth a look.

According to WWD, the game will also feature “embedded NFTs.” Though, that hardly sounds promising. The one-of-a-kind digital collectibles have gone from multi-million dollar auctions to freebies designed to promote throwaway pop culture, like Warner Bros. recent Space Jam reboot.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://www.engadget.com/louis-vuitton-mobile-game-nft-105507072.html?src=rss

Continue Reading

CNBC

Rocket Lab launches US Space Force satellite after its failed mission in May

Published

on

Rocket Lab has successfully launched a US military satellite in its first mission since its 20th Electron launch ended up in failure back in May. The company’s Electron rocket made its way to space from its New Zealand launch site, carrying a small demonstration satellite from the US Space Force called the Monolith. Peter Beck, Rocket Lab founder and CEO, tweeted shortly after the event that the payload was deployed and praised the team for a “flawless” launch.

Monolith’s target location is in low-Earth orbit around 370 miles above our planet. The satellite will demonstrate the use of a deployable sensor, “where the sensor’s mass is a substantial fraction of the total mass of the spacecraft, changing the spacecraft’s dynamic properties and testing ability to maintain spacecraft attitude control,” Rocket Lab explained in the official mission webpage. The results from the demonstration will help companies build more affordable satellites within shorter timeframes.

Rocket Lab’s 20th mission failed in May when it suffered an “anomaly” after the second stage ignition. The company lost both its Electron rocket and BlackSky’s Earth observation satellite payloads as a result of its second launch failure within the span of a year. After investigating the issue, Rocket Lab determined that the root cause was a problem with the second stage engine igniter system that “induced a corruption of signals within the engine computer.” The company says it has since implemented fixes “to prevent any future reoccurrence.”

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://www.engadget.com/rocket-lab-launches-us-space-force-satellite-103541725.html?src=rss

Continue Reading

AI

What The Original Wolf of Wall Street Thinks Of Bitcoin, Tether, Defi and the Crypto Space

Published

on

Although he made his fortune trading penny stocks and manipulating the markets, Jordan Belfort, the trader who inspired the iconic movie “The Wolf of Wall Street,” is currently not very happy with a lot of what’s going on with the crypto industry.

In a recent interview he explained in detail what excites him and what bothers him about the crypto space.

The Good: Bitcoin and DeFi

Belfort confesses to being an optimistic Bitcoiner. He explained that since discovering Bitcoin, he has found it to be an exciting idea.

He hopes that soon there will be more and better regulations for Bitcoin and the crypto industry. From his point of view, the more regulations there are, the better the consequences for cryptocurrency users and traders in the future.

“The sooner that massive regulation comes into the market, the better it is for Bitcoin, stablecoin and everything else. I think people are scared of regulations, they shouldn’t be. If you look back in history … let’s say the junk bond market, everybody said, ‘Oh no, the regulators are coming in! [But ultimately] the market got much bigger, so I think it’s a good thing,”

Belford commented that Bitcoin has a “bright future” and that the industry has evolved favorably over time, now being much more mature. He believes BTC could become a serious substitue for gold, and he expects the crypto to end the year somewhere between $45,000 and $70,000.


ADVERTISEMENT

Belfort is also very enthusiastic about DeFi, an industry he believes to be very innovative. He even advises a DeFi company today.

The Bad: Shitcoin ≠ Crypto

Belfort was concerned about the growing number of worthless cryptocurrencies being created to attract many unwary people who end up losing their money.

From his point of view, these scams are a harm to the crypto industry in general

There are so many scams, it is really sad … You have all these shitcoins out there. Literally, every single day, there are new ones being printed. They are simply designed to separate people from their money.

However, he assured that all crime leaves a trail and eventually the organizers of these scams will pay, even though they are currently walking free due to the legal loopholes.

He called for cleaning up the crypto markets, although he declined to specify which cryptocurrencies he considered to be scams.

The Ugly: Tether

For Belfort, the top stablecoin in the ecosystem at the moment is a stain on the crypto industry’s credibility. When asked if he considered Tether a scam, the seasoned trader stressed that he had several years warning about his suspicions regarding USDT.

I have been saying that since 2017 that I thought there was a big problem with Tether … It seemed like a pretty convenient thing that Tether supply expanded when it needed to expand. And then when they settled something civilly, I was like Wow, How did they get away with that?

He declined to call Tether a scam or a shitcoin. Still, He said that while Tether may currently operate cleanly, past mistakes could cost it dearly, even to the point of not being able to recover.

However, although this scenario could bring negative consequences for the crypto ecosystem, from his point of view, it would simply be a matter of time until the markets balance migrating to another stablecoin of proven legitimacy.

SPECIAL OFFER (Sponsored)

Binance Futures 50 USDT FREE Voucher: Use this link to register & get 10% off fees and 50 USDT when trading 500 USDT (limited offer).

PrimeXBT Special Offer: Use this link to register & enter POTATO50 code to get 50% free bonus on any deposit up to 1 BTC.

You Might Also Like:

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://coingenius.news/what-the-original-wolf-of-wall-street-thinks-of-bitcoin-tether-defi-and-the-crypto-space-12/?utm_source=rss&utm_medium=rss&utm_campaign=what-the-original-wolf-of-wall-street-thinks-of-bitcoin-tether-defi-and-the-crypto-space-12

Continue Reading

AI

What The Original Wolf of Wall Street Thinks Of Bitcoin, Tether, Defi and the Crypto Space

Published

on

Although he made his fortune trading penny stocks and manipulating the markets, Jordan Belfort, the trader who inspired the iconic movie “The Wolf of Wall Street,” is currently not very happy with a lot of what’s going on with the crypto industry.

In a recent interview he explained in detail what excites him and what bothers him about the crypto space.

The Good: Bitcoin and DeFi

Belfort confesses to being an optimistic Bitcoiner. He explained that since discovering Bitcoin, he has found it to be an exciting idea.

He hopes that soon there will be more and better regulations for Bitcoin and the crypto industry. From his point of view, the more regulations there are, the better the consequences for cryptocurrency users and traders in the future.

“The sooner that massive regulation comes into the market, the better it is for Bitcoin, stablecoin and everything else. I think people are scared of regulations, they shouldn’t be. If you look back in history … let’s say the junk bond market, everybody said, ‘Oh no, the regulators are coming in! [But ultimately] the market got much bigger, so I think it’s a good thing,”

Belford commented that Bitcoin has a “bright future” and that the industry has evolved favorably over time, now being much more mature. He believes BTC could become a serious substitue for gold, and he expects the crypto to end the year somewhere between $45,000 and $70,000.


ADVERTISEMENT

Belfort is also very enthusiastic about DeFi, an industry he believes to be very innovative. He even advises a DeFi company today.

The Bad: Shitcoin ≠ Crypto

Belfort was concerned about the growing number of worthless cryptocurrencies being created to attract many unwary people who end up losing their money.

From his point of view, these scams are a harm to the crypto industry in general

There are so many scams, it is really sad … You have all these shitcoins out there. Literally, every single day, there are new ones being printed. They are simply designed to separate people from their money.

However, he assured that all crime leaves a trail and eventually the organizers of these scams will pay, even though they are currently walking free due to the legal loopholes.

He called for cleaning up the crypto markets, although he declined to specify which cryptocurrencies he considered to be scams.

The Ugly: Tether

For Belfort, the top stablecoin in the ecosystem at the moment is a stain on the crypto industry’s credibility. When asked if he considered Tether a scam, the seasoned trader stressed that he had several years warning about his suspicions regarding USDT.

I have been saying that since 2017 that I thought there was a big problem with Tether … It seemed like a pretty convenient thing that Tether supply expanded when it needed to expand. And then when they settled something civilly, I was like Wow, How did they get away with that?

He declined to call Tether a scam or a shitcoin. Still, He said that while Tether may currently operate cleanly, past mistakes could cost it dearly, even to the point of not being able to recover.

However, although this scenario could bring negative consequences for the crypto ecosystem, from his point of view, it would simply be a matter of time until the markets balance migrating to another stablecoin of proven legitimacy.

SPECIAL OFFER (Sponsored)

Binance Futures 50 USDT FREE Voucher: Use this link to register & get 10% off fees and 50 USDT when trading 500 USDT (limited offer).

PrimeXBT Special Offer: Use this link to register & enter POTATO50 code to get 50% free bonus on any deposit up to 1 BTC.

You Might Also Like:

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.
Click here to access.

Source: https://coingenius.news/what-the-original-wolf-of-wall-street-thinks-of-bitcoin-tether-defi-and-the-crypto-space-13/?utm_source=rss&utm_medium=rss&utm_campaign=what-the-original-wolf-of-wall-street-thinks-of-bitcoin-tether-defi-and-the-crypto-space-13

Continue Reading
Blockchain4 days ago

Happy birthday Ethereum!

Cyber Security4 days ago

Android Banking Trojan Relies on Screen Recording and Keylogging Instead of HTML

Esports4 days ago

Tribes of Midgard Berserker: How to Unlock

Cyber Security4 days ago

How to Sell Cybersecurity

Investing5 days ago

How do you use top stock signals as a beginner?

Energy4 days ago

CATL anuncia a sua tecnologia inovadora mais recente ao lançar a sua primeira geração de baterias de íons de sódio

Esports4 days ago

Tribes of Midgard Berserker: How to Unlock

Energy4 days ago

CEMIG Geracão e Transmissão S.A. Announces Early Tender Date Results of its Cash Tender Offer for its 9.250% Senior Notes due 2024

Blockchain4 days ago

The GBA Annual Achievement Awards

AI4 days ago

Why Machine Vision Matters to Your Business

Energy4 days ago

Coding And Marking Equipment Market in Industrial Machinery Industry to grow by USD 1.45 billion | Technavio

Fintech4 days ago

Movers and Shakers: Finastra hires new payments lead, Credit Suisse onboards tech chief

Ecommerce3 days ago

How professional videos can help e-commerce brands attract more customers

AR/VR4 days ago

Competition: Win A Township Tale for Oculus Quest

Blockchain4 days ago

Binance to halt derivatives and futures trading in Europe

Cyber Security5 days ago

Guardicore Labs are Sharing Details of a Critical Vulnerability in Hyper-V

Energy4 days ago

CATL dévoile sa dernière technologie de pointe en lançant sa première génération de batteries sodium-ion

Esports4 days ago

Can You Play King’s Bounty 2 on PlayStation?

Fintech4 days ago

Chilean Fintechs Secure Millions; EBANX and Amazon Forge Payments Partnership

Esports4 days ago

Can You Play King’s Bounty 2 on PlayStation?

Trending