python爬虫入门(二)- python中的xpath的基本使用

Python XPath的基本用法

在Python中,XPath是一种用于查找和提取XML和HTML文档中特定元素的语言。它是一种强大而灵活的工具,可以帮助我们从复杂的文档中提取所需的数据。本文将详细介绍Python中XPath的基本用法,并提供示例代码。

安装lxml库

在使用XPath之前,我们需要安装lxml库。在命令行中运行以下命令来安装:

pip install lxml

导入相关模块

首先,我们需要导入相关的模块:requests用于获取网页内容,lxml用于解析网页内容并执行XPath查询。下面是一个示例:

import requests
from lxml import etree

获取网页内容

在使用XPath之前,我们需要获取网页的内容。可以使用requests库发送HTTP请求,并将响应的内容保存到变量中。下面是一个示例:

import requests

response = requests.get("https://www.fivcan.com")
html = response.text

在上面的示例中,我们使用get方法发送了一个GET请求,并将响应的内容保存在html变量中。

解析网页内容

使用lxml库的etree模块可以将网页内容解析为可执行XPath查询的对象。下面是一个示例:

from lxml import etree

tree = etree.HTML(html)

在上面的示例中,我们使用etree.HTML函数将html内容解析为一个ElementTree对象。

执行XPath查询

有了ElementTree对象,我们可以使用XPath语法执行查询。下面是一些常用的XPath查询示例:

# 获取所有的链接
links = tree.xpath("//a/@href")

# 获取所有的标题文本
titles = tree.xpath("//h1/text()")

# 获取第一个段落的文本
paragraph = tree.xpath("//p[1]/text()")

在上面的示例中,我们使用xpath方法执行了XPath查询。通过使用不同的XPath表达式,我们可以选择性地提取所需的元素或属性。

XPath的高级用法

在前面的博客中,我们已经介绍了Python中XPath的基本用法。本文将进一步探讨XPath的高级用法,包括谓语、属性选择、多路径选择和命名空间等。我们将提供详细的示例代码,并展开说明细节和注意事项。

谓语

XPath的谓语是一种条件筛选机制,可以根据特定的条件来选择元素。下面是一个示例:

from lxml import etree

html = """
<html>
  <body>
    <div class="container">
      <p>Paragraph 1</p>
      <p>Paragraph 2</p>
      <p>Paragraph 3</p>
    </div>
  </body>
</html>
"""

tree = etree.HTML(html)

# 选择第一个p元素
element = tree.xpath("//p[1]")
print(element[0].text)  # 输出:Paragraph 1

# 选择属性class为"container"的div元素下的所有p元素
elements = tree.xpath("//div[@class='container']/p")
for element in elements:
    print(element.text)

在上面的示例中,我们使用谓语选择了特定条件下的元素。通过在方括号中提供条件,我们可以根据元素的位置、属性或其他标识来筛选元素。

属性选择

XPath还支持根据元素的属性进行选择。下面是一个示例:

from lxml import etree

html = """
<html>
  <body>
    <div class="container">
      <p>Paragraph 1</p>
      <p>Paragraph 2</p>
      <p>Paragraph 3</p>
    </div>
    <a href="https://www.example.com">Link</a>
  </body>
</html>
"""

tree = etree.HTML(html)

# 选择所有带有href属性的a元素
elements = tree.xpath("//a[@href]")
for element in elements:
    print(element.attrib["href"])

在上面的示例中,我们使用属性选择器[@属性名]来选择具有特定属性的元素。通过attrib属性,我们可以获取元素的属性值。

多路径选择

XPath还支持多路径选择,可以在一个XPath表达式中选择多个路径下的元素。下面是一个示例:

from lxml import etree

html = """
<html>
  <body>
    <div>
      <p>Paragraph 1</p>
      <p>Paragraph 2</p>
    </div>
    <div>
      <p>Paragraph 3</p>
      <p>Paragraph 4</p>
    </div>
  </body>
</html>
"""

tree = etree.HTML(html)

# 选择两个div元素下的所有p元素
elements = tree.xpath("//div/p")
for element in elements:
    print(element.text)

在上面的示例中,我们使用斜杠/将两个路径连接起来,从而选择多个路径下的元素。这样可以方便地选择多个相关路径下的元素。

命名空间

如果要处理包含命名空间的XML文档,需要在XPath表达式中考虑命名空间的前缀。下面是一个示例:

from lxml import etree

xml = """
<root xmlns:ns="http://www.example.com">
  <ns:element>Value</ns:element>
</root>
"""

tree = etree.XML(xml)

# 使用命名空间前缀选择元素
element = tree.xpath("//ns:element", namespaces={"ns": "http://www.example.com"})
print(element[0].text)  # 输出:Value

当然!除了前面提到的谓语、属性选择、多路径选择和命名空间,XPath还有一些其他高级用法。下面我们将继续介绍这些用法,并提供示例代码。

文本提取

XPath不仅可以选择元素,还可以提取元素内部的文本内容。下面是一个示例:

from lxml import etree

html = """
<html>
  <body>
    <div class="container">
      <p>Paragraph 1</p>
      <p>Paragraph 2</p>
      <p>Paragraph 3</p>
    </div>
  </body>
</html>
"""

tree = etree.HTML(html)

# 提取第一个p元素的文本内容
text = tree.xpath("//p[1]/text()")
print(text[0])  # 输出:Paragraph 1

在上面的示例中,我们使用text()函数来提取元素内部的文本内容。通过索引获取列表中的第一个元素,即可获得所需的文本内容。

属性值提取

除了提取文本内容,XPath还可以提取元素的属性值。下面是一个示例:

from lxml import etree

html = """
<html>
  <body>
    <a href="https://www.example.com">Link</a>
  </body>
</html>
"""

tree = etree.HTML(html)

# 提取a元素的href属性值
href = tree.xpath("//a/@href")
print(href[0])  # 输出:https://www.example.com

在上面的示例中,我们使用@属性名的语法来提取元素的属性值。通过索引获取列表中的第一个元素,即可获得所需的属性值。

运算符使用

XPath还支持多种运算符,可以用于比较和筛选元素。下面是一些常用的运算符示例:

from lxml import etree

html = """
<html>
  <body>
    <div class="container">
      <p>Paragraph 1</p>
      <p>Paragraph 2</p>
      <p>Paragraph 3</p>
    </div>
  </body>
</html>
"""

tree = etree.HTML(html)

# 选择文本长度大于5的p元素
elements = tree.xpath("//p[string-length(text()) > 5]")
for element in elements:
    print(element.text)

在上面的示例中,我们使用运算符>来比较元素的文本长度。这样可以根据特定的条件筛选出所需的元素。

轴(Axis)使用

XPath的轴(Axis)允许我们在文档中沿着特定的方向选择元素。下面是一个示例:

from lxml import etree

html = """
<html>
  <body>
    <div>
      <p>Paragraph 1</p>
      <p>Paragraph 2</p>
    </div>
    <div>
      <p>Paragraph 3</p>
      <p>Paragraph 4</p>
    </div>
  </body>
</html>
"""

tree = etree.HTML(html)

# 选择第一个div元素下的所有p元素
elements = tree.xpath("//div[1]/p")
for element in elements:
    print(element.text)

在上面的示例中,我们使用轴[]来选择第一个div元素下的所有p元素。这样可以沿着特定的方向选择元素。

注意事项

  • XPath表达式是一种强大而灵活的工具,但也需要一定的学习和经验才能熟练使用。建议参考XPath的相关文档和教程,深入理解其语法和特性。
  • 当使用XPath查询时,要确保提供的XPath表达式准确匹配目标元素。否则,可能会返回错误的结果或无法找到所需的元素。
  • 在处理HTML或XML文档时,有时需要注意文档结构的复杂性和变化性。可能需要根据实际情况调整XPath表达式以适应不同的文档结构。

下面是常用xpath功能大全,我做成了表格,可供参考

1.节点

表达式描述
nodename选取此节点的所有子节点。
/从根节点选取。
//从匹配选择的当前节点选择文档中的节点,而不考虑它们的位置。
.选取当前节点。
..选取当前节点的父节点。
@选取属性。

2.谓语

路径表达式结果
/bookstore/book[1]选取属于 bookstore 子元素的第一个 book 元素。
/bookstore/book[last()]选取属于 bookstore 子元素的最后一个 book 元素。
/bookstore/book[last()-1]选取属于 bookstore 子元素的倒数第二个 book 元素。
/bookstore/book[position()<3]选取最前面的两个属于 bookstore 元素的子元素的 book 元素。
//title[@lang]选取所有拥有名为 lang 的属性的 title 元素。
//title[@lang='eng']选取所有 title 元素,且这些元素拥有值为 eng 的 lang 属性。
//title[not(@lang='eng')]取反,取lang不等于'eng'的。
//title[@lang='eng' and @class='test']与为and,或为or。
/bookstore/book[price>35.00]选取 bookstore 元素的所有 book 元素,且其中的 price 元素的值须大于 35.00。
/bookstore/book[price>35.00]/title选取 bookstore 元素中的 book 元素的所有 title 元素,且其中的 price 元素的值须大于 35.00。
//title | //price选取文档中的所有 title 和 price 元素。

3.轴

轴名称表达式描述
ancestorxpath('./ancestor::*')选取当前节点的所有先辈节点(父、祖父)
ancestor-or-selfxpath('./ancestor-or-self::*')选取当前节点的所有先辈节点以及节点本身
attributexpath('./attribute::*')选取当前节点的所有属性
childxpath('./child::*')返回当前节点的所有子节点
descendantxpath('./descendant::*')返回当前节点的所有后代节点(子节点、孙节点)
followingxpath('./following::*')选取文档中当前节点结束标签后的所有节点
following-siblingxpath('./following-sibling::*')选取当前节点之后的兄弟节点
preceding-siblingxpath('./preceding-sibling::*')选取当前节点之前的兄弟节点
parentxpath('./parent::*')选取当前节点的父节点
precedingxpath('./preceding::*')选取文档中当前节点开始标签前的所有节点

4.功能函数

函数用法解释
starts-withxpath('//div[starts-with(@id,"ma")]')选取id值以ma开头的div节点
contains()xpath('//div[contains(@id,"ma")]')选取id值包含ma的div节点
text()xpath(//a[text()="下一页"]')选取节点文本包含ma的div节点
not()//title[not(@lang='eng')]取反,取lang不等于'eng'的

5.复杂查询

用法解释
//tr[contains(./td/label/text(), "2023")]内部label包含2023的所有tr
//span/*[self::p or self::h3]span下的p或h3

总结

本文介绍了Python中XPath的更多高级用法,包括文本提取、属性值提取、运算符使用和轴使用。XPath是一种强大而灵活的工具,可以帮助我们处理复杂的文档并提取所需的数据。希望本文对你进一步了解和使用Python中的XPath有所帮助。

以上就是关于Python XPath的更多高级用法的详细说明。谢谢阅读!

参考资料:

36 Replies to “python爬虫入门(二)- python中的xpath的基本使用”

  1. I’ve been surfing online more than 2 hours today, yet I never found any interesting article like yours. It’s pretty worth enough for me. In my view, if all website owners and bloggers made good content as you did, the internet will be much more useful than ever before.
    I’ll immediately grab your rss as I can not find your e-mail subscription hyperlink or e-newsletter service. Do you have any? Please allow me know so that I may subscribe. Thanks.
    https://sinee-nebo-golova.com

  2. Awsome article and right to the point. I am not sure if this is really the best place to ask but do you guys have any ideea where to hire some professional writers? Thank you 🙂

  3. F*ckin’ awesome things here. I am very happy to peer your post. Thank you a lot and i’m having a look ahead to touch you. Will you kindly drop me a e-mail?

  4. Youre so cool! I dont suppose Ive read anything like this before. So good to seek out anyone with some authentic ideas on this subject. realy thanks for beginning this up. this website is something that’s needed on the internet, somebody with a bit originality. helpful job for bringing something new to the web!

  5. Thanks for sharing excellent informations. Your web-site is very cool. I am impressed by the details that you have on this website. It reveals how nicely you perceive this subject. Bookmarked this web page, will come back for more articles. You, my friend, ROCK! I found just the info I already searched all over the place and simply could not come across. What a great website.

  6. https://www.datingwise.com/review/flirt.com/

    Flirt.com is for adults looking for fun, flirty encounters rather than serious dating. The site is aimed at the younger crowd, though there are older members there and some seeking longer term relationships. It offers free membership for women, while men can join for free but must pay for additional services such as email.
    Flirt has been recently revamped and is designed for people looking for casual dating. Most people there are in their twenties and early thirties, though there is no upper age limit. It’s owned by the Cupid Dating network and caters mostly to members in the UK, the U.S. and Australia, though membership is open to anyone.
    Flirt is “spicier” than your regular dating site – don’t expect to find your next significant other there.
    Naughty mode
    This site is designed to have a light, fun feel to it. It is not intended to be an “adult” site, though there is some mature content. Most adult content can be blocked by switching-off “naughty mode” (the initial setting).
    This will hide any images that are explicit. Flirt is a worth a look if you are single and looking to meet new people and have a little fun. Those looking for more serious relationships would probably be better off looking elsewhere.
    Features
    Flirt.com is feature rich, offering email, message boards, chat rooms, member diaries, videos as well as basic flirts and emails. Flirt has a dedicated mobile site for those wanting access their matches on the go. They also sponsor speed dating and other live events for those who want to meet someone in person.
    Membership
    Women have access to all features of for free. Men can join for free, but will need a paid membership in order to use some features of the site. Despite being free for women there is still a very high proportion of male users.
    Flirt ist ok. A few fake profiles (like everywhere), a few cam girls (like everywhere) and a few scammers (like everywhere) but generally the site seems to be real. Personally prefer because i’ve actually hooked up twice using it, but just wanted to try somethin’ new so decided to give Flirt a chance.
    Avatar
    topiJuly 11th, 2017
    Useless
    Not worth it. I had no luck on this site after six months.
    Avatar
    freedFebruary 8th, 2017
    Just average
    how do i join?
    Avatar
    ChrisOilSeptember 12th, 2016
    Recommended
    Though it’s a casual site, I met my love here. So everything’s in your hands. Try, you won’t lose anything.
    Avatar
    Paul87July 18th, 2016
    Above average
    Good site for flirting and one night stands! Unfortunately one day this won’t be enough for you and flirt cannot offer you something serious.
    Avatar
    Craig8686904June 21st, 2016
    Recommended
    I’m really glad that a friend of mine gave me the advice to register on Flirt to make my life more spicy. I wasn’t really going to have anything more than just a naughty chat but it turned out that there’s a nice lady in mt city who’s willing to date with me. I’m freaking happy now
    Avatar
    Chris53June 13th, 2016
    Recommended
    Though it’s a casual site, I met my love here. So everything’s in your hands. Try, you won’t lose anything.
    Avatar
    RobertGreen84June 8th, 2016
    Above averag
    Cute looking site like many others however only here I’ve had 5 dates within 3 weeks after the start. Also I should note that flirt sometimes really hard to use and it’s taking some time to feel yourself comfortable during usage of it and actually it’s not because of gliches or something simply the pictures of buttons are obvious so sometimes you can find yourself on the page you haven’t wanted to open. However I should admit that in the end it worth all the troubles in the start.
    Avatar
    JohnMApril 13th, 201
    Avoid
    This site is bull****, it’s a total scam, the profiles of women are not even real they are all fake, when you create a profile and it becomes active they suck you in by sending you lots of messages and winks from so called women which are not even real and don’t actually exist and because you can’t read the messages as an unpaid member to be able to read the messages you have to subscribe and pay for a membership then once you do that and you respond to the messages you don’t get a reply back.
    It’s a con, warning to other people out there do not join this site.
    Avatar
    Michael_4302January 15th, 201
    Useless
    This site claims that singles are in your area, but in truth that they live elsewhere. I had received a lot of mail from people that the site claimed were in my area, but they actually lived far away. Beware of scammers as well, I have found quite a bit of them on this whose profiles seemed to be processed quickly since their information is available. However, those members who may actually be real usually have the contents of their profile information pending. I’ve seen a lot of scam activity on this site and very little actual people.

  7. Thanks for the marvelous posting! I actually enjoyed reading it, you happen to be a great author.I will be sure to bookmark your blog and definitely will come back in the foreseeable future. I want to encourage you to definitely continue your great posts, have a nice day!

  8. Hi! Someone in my Facebook group shared this website with us so I came to check it out. I’m definitely loving the information. I’m bookmarking and will be tweeting this to my followers! Terrific blog and terrific design and style.

  9. What’s Going down i am new to this, I stumbled upon this I have discovered It absolutely useful and it has aided me out loads. I am hoping to give a contribution & assist other users like its aided me. Good job.

  10. 120 Flirt com Reviews | flirt.com @ PissedConsumer
    Flirt com has 120 reviews (average rating 1.1). Consumers say: Unauthorized money taken out of my account, It’s a $1.99 for one day drive I paid it you all took $5.99 out of my account two times and then took a extra $23.31 out of my account this is outrageous I did not ask for that I want my money… Flirt com has 1.1 star rating based on 41 customer reviews. Consumers are mostly dissatisfied Rating Distribution 95% negative 2% positive Pros: Everyone is fake, I have no idea, I-o nu folosesc. Cons: Scamming, All fake profile there, All kind. Recent recommendations regarding this business are as follows: “Don’t activate anything on this fake, fony and complicated site app.”, “Nothing to recommend at this stage”, “Do not use this site”, “Don’t pay for this scam it’s a SCAM”, “Do not have any type of business with this fraudulent company”. Most users ask Flirt com for the refund as a solution to their issues. Consumers are not pleased with Discounts and Special Offers and Diversity of Products or Services. The price level of this organization is high according to consumer reviews.They charged my debit card without authorization and continuously stated that it was 100% free, regardless of how I searched. This was unprofessional. I was very disappointed in every site I went to, stating that it was free.They took money in m’y bank account when.im.not on this site since febuary,i canceld and they took 47.90$ I wanna talk to.someone but when i call number not Unauthorized money taken out of my account It’s a $1.99 for one day drive I paid it you all took $5.99 out of my account two times and then took a extra $23.31 out of my account this is outrageous I did not ask for that I want my money back in Updated by user Jan 21, 2024
    You can just take money out of those people’s account without them giving you permission to I want my money back I was only supposed to send me one dollar 99 cent no more no less I want my money thank you Original review Jan 21, 2024 False promotion you made me pay $36 for something that I thought I was paying $99 sent for for one day I did not want it for longer than one day I was just being curious and you stole everything in my account and I want it back thank you for understanding and putting my belongings back to the rightful beneficiary I’m trying to Deactivate this account. Every number you call is not on service. That tells a person automatically this is a *** company. Good thing I didn’t pay. Nonetheless, Unsubscribe me, Deactivate/Disqualify my account at once! Crazy part is, they know exactly what they’re doing! Pretending not to pay a person any mind. Don’t send me any emails, text messages, etc. Except to communicate that my account is done for. Thank you for your anticipated cooperation. You Bums! Preferred solution: Deactivate/Delete/Disqualify my account and picture. Do not use my private information at all. User’s recommendation: Don’t activate anything on this fake, fony and complicated site app. They charged my debit card without authorization and continuously stated that it was 100% free, regardless of how I searched. This was unprofessional. I was very disappointed in every site I went to, stating that it was free.This company is a scam they told me to pay $199 for a one-day pass and they charged me $5.99 two times and 23.31 time that’s $35 you cannot take money out of people’s account without telling them I ag I would like for my whole refund to be put back into my cash app account y’all had no reason and no business taking $35 out of my account and I only subscribe for $1.99 y’all took $5.99 out of my account two times in $23.31 out of my account once this is uncalled for and unnecessary and it is bad promotion please return my property back to the rightful beneficiary thank you Attempting to debit me after several messages that i dont have knowledge about the platform
    May 24, 2024 I lost my cross bag containing my phone, ATM card and some valid ID card about two months ago or there about. When I eventually find it, I discovered that flirttender.com has been making attempt to debit me. I sent severally messages that I don’t know what the platform is all about neither do I have knowledge about it but to no avail. This debit attempt is getting much and I will like you to unsubscribe me or remove my bank information from your platform and please stop the attempt of debiting me. I never initiated this and I don’t know who use my Information for that purpose. Please, kindly do something to stop this. Kindly find the account details; 535522****694369 Moses Oyinloye Chatting but no actions . Stay away. ,…andmd….h&huhhhhhygfffdddfcfdffcffddrdddzddddxddddr c d dr Pros Too much chat with paid texters
    Cons: Too much chat with paid texters Preferred solution: Full refund My teenage son, I got my card and used it on this site.I called him and asked him to please.Uh, refund me my money and to take my card off of this site.They did, and now they’re trying to charge my card again.For a monthly fee This site needs to be shut down and the police need to be involved in this this is ridiculous. This site is a scam I am being charged every month since I have used Flirty.com a couple of months ago. All I was paying for were credits to chat with other customers.I never knew that they were going to charge me a monthly service. What is the service for? All I needed were credits to use their services. So why are they charging me $12.99 a month? I’m not getting any credits for it. If that’s not included, then what is the $12.99 monthly charge for? I tried to call them today, and the phone number 888-884-**** is no longer in service. How can I get my credit card company to stop paying for this monthly charge that I didn’t sign up for? Preferred solution: I want my subscription cancel asap. I have no money in my account, but they are still allowed to charge me ? I have been trying to cancel the prescription for this site for 4 days the number that they give me the 884 **** does not work I’ve been trying to cancel it either saying you’re going to keep billing me I want to cancel the subscription but it just will not allow me to cancel it I talked to a consumer representative they said they were going to cancel it but they said they were going to give me a free month I said I didn’t need it but they said it’s free just use it we’re sorry about the inconvenience now I can’t I’m getting billed again for it I need to cancel this thing I’m not going to keep paying you guys man I’m not going to do it Needed to talk about £250 withdrawn from ROYAL BANK OF SCOTLAND, without my permission Took money without any explanation and I sent you an email that I wanted to cancel everything with you please cancel everything and delete my account I sent an email two days ago that I wanted to cancel everything and some how today I woke up and saw that you took money from my bank account with my right I want you to cancel the transaction you took and cancel everything I have on your page because I wanted from the first day to cancel everything so play CANCEL EVERYTHING I DONT WANNA SEE ANYTHING TAKING FROM MY CARD FROM YOUR COMPANY OR ELSE I WILL MAKE TO TELL MY LAWYER SND MAKE SOMETHING FROM REAL PLEASEEEEE CANCEL EVERYTHING WITH MY ACCOUNT

  11. Hello, Neat post. There is a problem with your site in web explorer, may check this… IE nonetheless is the marketplace chief and a good part of folks will leave out your fantastic writing due to this problem.

  12. The next time I read a blog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my choice to read, but I actually thought youd have something interesting to say. All I hear is a bunch of whining about something that you could fix if you werent too busy looking for attention.

  13. I cherished as much as you’ll obtain carried out right here. The cartoon is attractive, your authored material stylish. nonetheless, you command get bought an edginess over that you want be handing over the following. sick definitely come more previously again as precisely the same nearly a lot often inside of case you defend this increase.

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注