Education items are duplicated

[hugoliv]

Hi,

This package is working great.

Howerver, education field seems to not be well populated with many duplicates.

Here is an example with a profile containing 3 different education items but only the first one is scraped and duplicated 3 times:

[Education(institution_name='Université Paris Cité', linkedin_url='https://www.linkedin.com/company/19143575/', website=None, industry=None, type=None, headquarters=None, company_size=None, founded=None, from_date='2008', to_date='2012', description='', degree='Master 1, Psychologie clinique'), Education(institution_name='Université Paris Cité', linkedin_url='https://www.linkedin.com/company/19143575/', website=None, industry=None, type=None, headquarters=None, company_size=None, founded=None, from_date='2008', to_date='2012', description='', degree='Master 1, Psychologie clinique'), Education(institution_name='Université Paris Cité', linkedin_url='https://www.linkedin.com/company/19143575/', website=None, industry=None, type=None, headquarters=None, company_size=None, founded=None, from_date='2008', to_date='2012', description='', degree='Master 1, Psychologie clinique')]

[hugoliv]

Well, the issue is fixed with this get_education function :

   def get_educations(self):
    url = os.path.join(self.linkedin_url, "details/education")
    self.driver.get(url)
    self.focus()
    main = self.wait_for_element_to_load(by=By.TAG_NAME, name="main")
    self.scroll_to_half()
    self.scroll_to_bottom()
    main_list = self.wait_for_element_to_load(name="pvs-list__container", base=main)

    seen_educations = set()  # Pour éviter les doublons

    for position in main_list.find_elements(By.CLASS_NAME, "pvs-list__paged-list-item"):
        try:
            position = position.find_element(By.CSS_SELECTOR, "div[data-view-name='profile-component-entity']")
            elements = position.find_elements(By.XPATH, "*")
            if len(elements) < 2:
                continue

            institution_logo_elem = elements[0]
            position_details = elements[1]

            try:
                institution_linkedin_url = institution_logo_elem.find_element(By.XPATH, "*").get_attribute("href")
            except NoSuchElementException:
                institution_linkedin_url = None

            position_details_list = position_details.find_elements(By.XPATH, "*")
            position_summary_details = position_details_list[0] if len(position_details_list) > 0 else None
            position_summary_text = position_details_list[1] if len(position_details_list) > 1 else None

            if not position_summary_details:
                continue

            outer_positions = position_summary_details.find_element(By.XPATH, "*").find_elements(By.XPATH, "*")

            institution_name = outer_positions[0].find_element(By.TAG_NAME, "span").text.strip() if outer_positions else ""
            degree = outer_positions[1].find_element(By.TAG_NAME, "span").text.strip() if len(outer_positions) > 1 else None

            from_date = None
            to_date = None

            if len(outer_positions) > 2:
                try:
                    times = outer_positions[2].find_element(By.TAG_NAME, "span").text.strip()
                    if "-" in times:
                        parts = [p.strip() for p in times.split("-")]
                        from_date = parts[0] if len(parts) > 0 else None
                        to_date = parts[1] if len(parts) > 1 else None
                except (NoSuchElementException, ValueError):
                    pass

            description = position_summary_text.text.strip() if position_summary_text else ""

            # Générer une clé unique pour éviter les doublons
            unique_key = f"{institution_name}|{degree}|{from_date}|{to_date}"
            if unique_key in seen_educations:
                continue
            seen_educations.add(unique_key)

            education = Education(
                from_date=from_date,
                to_date=to_date,
                description=description,
                degree=degree,
                institution_name=institution_name,
                linkedin_url=institution_linkedin_url
            )
            self.add_education(education)

        except (NoSuchElementException, IndexError):
            continue

If someone want to create a pull request...

[stickerdaniel]

Can you provide the profile where you encountered this issue? I couldn't replicate this issue

[hugoliv]

Hi,
Hum, it happens with this one for example "https://www.linkedin.com/in/chris-hiverlet-ba4b4638/".

[syed0596]

@stickerdaniel attached the same problem and fixed it: #254

[stickerdaniel]

If all you need is profile scraping, this rewrite I started might be useful to you. https://github.com/stickerdaniel/fast-linkedin-scraper It is using playwright instead of selenium for better performance, has improved value parsing, and a simple api.

[syed0596]

That's helpful you gained a follower, although I tweaked this one for better performance too in my local repo. Follow for an update. 🙂