Gender Gap in Wikipedia’s content

Only 15% of all biographies on English Wikipedia belong to women. Women and men are portrayed differently on Wikipedia in terms of article structure, the use of infoboxes, network properties, notability etc. This research project is aimed at mapping the gender gap on Wikipedia in terms of its content. This work is done as a followup report to my presentation at WikiWomenCamp 2017. The aim is to create a review of peer-reviewed research papers on gender gap on Wikipedia’s content.

Methods

• Find all relevant articles for the analysis using Google scholar. Keywords used are ‘Wikipedia’, ‘gender’, ‘content’, ‘women’, ‘bias’ and various relevant combinations of these words.
• Screen the title and abstract to include only those studies that fit the inclusion criteria. Further screening for content to only include the studies about gender gap in Wikipedia’s content.
• Assess the validity and reliability of the results
• Systematic presentation of the findings

Rosie Stephenson Goodnight has worked extensively on bridging the gender gap in English Wikipedia. Photo: Camelia.boban, CC-BY-SA 4.0, Wikimedia Commons

Results

The results were summerized under four categories :

• Coverage bias : Coverage bias occurs when men and women are covered differently on Wikipedia. For example, the coverage bias may manifest as differences in the number of notable women and men portrayed on Wikipedia.

Research	Data	Methods	Findings
Wagner et al [1]	Wikipedia in 6 language editions	Wikipedia in 6 languages compared to several datasets: Freebase, Pantheon, Human Accomplishment, crawled the content of articles about people in the reference datasets using Wikipedia’s API (November 2014).	Men and women are covered equally well on Wikipedia and articles about women tend to be longer than articles about men on Wikipedia, when compared to those from the reference datasets.
Graells-Garrido et al [2]	The DBPedia 2014 dataset, The Wikipedia English Dump of October 2014	The DBPedia and Wikipedia data dump were analysed for metadata properties. The gender of a biography, whenever not mentioned, was determined by ‘inferred gender for Wikipedia biographies’ (Bamman and Smith)	15% of articles in ‘Person class’ were about women. In comparison to the global proportion of women, the categories that over-represent women are Artist, Royalty, FictionalCharacter, Noble, BeautyQueen, and Model.
Reagle & Rhue [3]	Biographical subjects from several sources (100 Most Influentiial figures in American History, TIME magazine’s list of 2008’s most influential people, Chambers Biographical dictionary, American National Biography Online) compared to English Wikipedia and Britannica.	A Python program was used to compare web pages related to the subjects targeted in the reference sources. Google API was queried for top four results. Gender was guessed by the balance of gendered pronouns (she, her, he, his). The length of an article is determined by the words of article content and does not include citations and other miscellany.	Wikipedia provides better coverage and longer articles on women than Britannica. Wikipedia has more articles about women than Britannica in absolute terms, but articles about women on Wikipedia are more likely to be missing than articles about men compared to Britannica.
Wagner et al [4]	DBPedia 2014 dataset, inferred gender for Wikipedia bios	Calculated the number of language editions in which per biography is represented and google search volume of women’s bio, compared them with Wikipedia articles	Women in Wikipedia are more notable than men, which the authors interpret as the outcome of a subtle glass ceiling effect.

• Structural bias : Structural bias refers to preferential use of gender-specific tendencies while connecting articles on notable people. For example, there may be more links to men’s biographies on articles related to women.

WikiGap is a program dedicated to closing the content gap on Wikimedia.

Research	Data	Methods	Findings
Wagner et al [1]	Wikipedia in 6 language editions	Wikipedia’s API (November 2014), analysed for probability that a link from article with gender g1 ends in an article with gender g2.	Articles about women connect less to articles about men via interlinks. Articles about people with the same gender tend to link to each other. Articles about women tend to link more to articles about men than the opposite. Men are more central than women in English, Russian and German language Wikipedia.
Graells-Garrido et al [2]	The DBPedia 2014 dataset, The Wikipedia English Dump of October 2014	Proportion of links from gender to gender was calculated and tested against expected proportions. Analysed distribution of PageRank by gender to understand centrality.	Women biographies tend to link more to other women than to men. The article with highest centrality tend to be predominantly about men, beyond what one could expect from the structure of the network.
Wagner et al [4]	DBPedia 2014 dataset, inferred gender for Wikipedia bios, attributes, PageRank	Explored to what extent the connectivity between people is influenced by gender. Investigated the relation between the centrality of people and their gender using PageRank.	The top-ranked women according to PageRank are slightly less central than men, and the centrality of women decreases faster than that of men with decreasing rank. There exists a bias in the generation of links by Wikipedia editors, favoring articles about men.

• Lexical bias : Lexical bias refers to the inequalities in the terms used to describe men and women on Wikipedia. For example, the articles about women are more likely to have details about their family life.

Research	Data	Methods	Findings
Wagner et al [1]	Wikipedia in 6 language editions	Open vocabulary approach where classifier determines which words are most effective in distinguishing the gender of the person an article is about. Log likelihood ratios are used for comparing different feature-outcome relationships.	There is lower salience of male-related words in articles about men, which can be related to the idea of male as the null gender (there is a social bias to assume male as the standard gender in certain social situations). Words like “married”, “divorced”, “children” or “family” are much more frequently used in articles about women. This study confirms that men and women are presented differently on Wikipedia and that those differences go beyond what we would expect due to the history of gender inequalities.
Graells-Garrido et al [2]	The DBPedia 2014 dataset, The Wikipedia English Dump of October 2014, Linguistic Inquiry and Word count (LIWC) dictionary	To explore which words are more strongly associated with each gender, Pointwise Mutual Information is measured over the set of vocabulary in both genders. Also considered burstiness, a measure of word importance in a single document according to the number of times it appears within the document, under the assumption that important words appear more than once (they appear in bursts) when they are relevant in a given document.	Marriage and sex-related content are more frequent in women’s biographies and cognition related content is highlighted in men’s biographies. Words most associated with men are mostly about sports, while the words most associated with women are to arts, gender and family. Of particular interest are two concepts strongly associated with women: her husband and first woman.
Wagner et al [4]	Overview of English Wikipedia biographies, inferred gender for Wikipedia bios	Analysed gender topic, relationship topic and family topic in Wikipedia’s biographies. Quantified the tendency of expressing positive and negative aspects of biographies with adjectives, as a measure of the degree of abstraction of positive and negative content.	Family-, Gender-, and relationship-related topics are more present in biographies about women, linguistic bias manifests in Wikipedia since abstract terms tend to be used to describe positive aspects in the biographies of men and negative aspects in the biographies of women.

• Visibility bias: Visibility bias occurs when articles related to men and women are differently promoted within Wikipedia. For example, men’s biographies are potentially more likely to be featured articles than women’s biographies, although the difference is not significant.

Research	Data	Methods	Findings
Wagner et al [1]	Wikipedia in 6 language editions	Proportion of women’s biographies that make it to the main page of Wikipedia	Selection procedure of featured articles of Wikipedia community does not suffer from gender bias.

 

Women Wikimedians at WikiConference, India 2016. According to a 2011 survey, only 3% of Indian Wikimedians were women. Photo: Afifa Afrin, CC-BY-SA, Wikimedia Commons

References

1. ↑ ^a b c d It’s a man’s Wikipedia? Assessing Gender Inequality in an online Encyclopedia Wagner, Claudia; Garcia, David; Jadidi, Mohsen; Strohmaier, Markus (May 2015). “It’s a man’s Wikipedia? Assessing Gender Inequality in an online Encyclopedia”. Proceedings of the Ninth International AAAI Conference on Web and Social Media. Retrieved 28 July 2017.
2. ↑ ^a b c Graelles-Garrido, Eduardo; Lalmas, Mounia; Menczer, Filippo (September 2015). “First Women, Second Sex : Gender Bias in Wikipedia”. Social and Information Networks. Retrieved 28 July 2017.
3. ↑ Reagle, Joseph; Rhue, Lauren (2011). “Gender bias in Wikipedia and Britannica”. International Journal of Communication S: 1138–1158. Retrieved 28 July 2017.
4. ↑^a b c Wagner, Claudia; Graelles-Garrido, Eduardo; Garcia, David; Menczer, Filippo (2016). “Women through the glass ceiling: gender asymmetries in Wikipedia” (PDF). EPJ Data Science. Retrieved 30 July 2017.

 

The same article can be found on meta-wiki here. A longer presentation containing information regarding gender gap research on Wikipedia can be found here.

Featured photo courtesy: Martina Cora, CC-BY-SA 4.0, Wikimedia Commons

Women at Wikimania 2013

Wikimania 2013, the annual conference of the Wikimedia movement, had the participation of more than 60 women. As of July 30, women accounted 20 percent of online registrations for Wikimania 2013. There was a separate track for ‘Women in Wikimedia’ on Day 2 of the conference. Around 40 women participated in WikiWomen’s Luncheon which happened on the same day.

Organizing team

The Program Committee  of 11 comprised of two women, Katie Filbert and Sarah Stierch. Ellie Young facilitated and supported the organizing team in her capacity as the conference co-ordinator of Wikimedia Foundation. Katie Chan was a member of the scholarship committee of Wikimania 2013.

Keynote by Sue Gardner

The keynote on the final day of the conference was delivered by Sue Gardner, the Executive Director of Wikimedia Foundation. In response to a question from the press, she replied : “I wish we had solved the (gender gap) problem (in Wikimedia), but we didn’t.”

WikiWomen’s Lunch during Wikimania-2013. Sue Gardner,CC-BY-SA.

Wikiwomen’s Luncheon

Wikiwomen’s Luncheon , the luncheon for women attendees of Wikimania 2013, was held on the second day of the conference. Around 40 women participated in the luncheon. Conversation was facilitated by Sue Gardner. Sue told that the participation in the Wikiwomen’s Lunch has rose from 11 in Taipei, 2011 to more than 100 in Washington D.C, 2012.  Gardner observed that when Wikimedia’s editor community is dominated by educated males, and expansion is by word-of-mouth, it will not “naturally grow to be as diverse as it otherwise could have been.”Sarah Stierch, the Program Evaluation Community Coordinator for the Wikimedia Foundation, shared her experiences about volunteering with the Wikimedia Foundation. Staff members of Wikimedia Deucheland passed information and distributed flyers of their upcoming Diversity Conference , which is scheduled to take place in Berlin in November.

Women speakers

Sue Gardner at Wikimedia 2013. By Lvova [CC-BY-SA-3.0], via Wikimedia Commons

 

Talks, panels, pre-conference events and workshops by women speakers were:

1. Open Street Map Workshop (Katie Filbert)
2. Women and non-conventional education – a study from Indian cultural context  (Kavya Manohar)
3. Growing the Arabic Wikipedia through the Wikipedia Education Program  (LiAnna Davis)
4. Encouraging the creation and development of articles about women in Ibero-America (Ivana Lysholm)
5. The coolest projects of Wikimedia Chapters – be inspired (Nicole Ebber; together with Lodewijk Gelauff)
6. Idea Lab Brainstorm (Siko Bouterse & Heather Walls)
7. Dev Camp (Sumana Harihareswara and others)
8. Promoting diversity in the German Wikipedia (Ilona Buchem)
9. Towards bridging the gender gap in Indian Wikimedia Community (Jadine Lannon & Netha Hussain)
10. Bridging the gender gap with women scientists (Emily Temple Wood)

Women participants in panel discussions were :

1. Carmen Alcázar and Monica Mora in Wiki Loves Monuments
2. Sumana Harihareswara in Transparency and Collaboration in Wikimedia Engineering

(This is an incomplete list. If you know a woman speaker at Wikimania 2013, feel free to tell me to add her name here)

Press

1. “Wikipedia fails to bridge gender gap“ (South China Morning Post, 11 August 2013) by Keira Huang

2. “Women contributors still face hurdles at Wikipedia“ (The Wall Street Journal, 19 August 2013) by Riva Gold

Outputs from Ada Camp

It has been around eight months since I traveled to the U.S for the first time to attend Ada Camp D.C. Looking back, I find the Ada Camp as one of the most fulfilling experiences I had ever had.

The Ada initiative is a non-profit organization dedicated to increasing the participation and status of women in open technology and culture. They organize the Ada Camp and other women’s hangouts in different cities of the world. The one I attended was the second Ada Camp held in Washington D.C, U.S.A.

I had to appear for my exams soon after the camp, and I was skeptical if I would get a U.S visa. I was the only attendee traveling from India. I would miss my classes at the University for a week, and I knew that catching them up would be hard. Despite all these, I decided that I should attend the Ada Camp anyway. I guess I was lucky, because I got the visa without much hassle. I was granted leave from college. I couldn’t believe that I would be flying to join the Ada Campers in Washington D.C!

The Ada Camp brought together more than 100 enthusiastic women from all over the world. The attendees were a diverse mix of individuals, homemakers, mommies, engineers, researchers, students, social media analysts and many other people from different backgrounds. The participants were from different nationalities, but all of them were driven into applying for the Ada Camp because of their sheer love for open source stuff. I met many women who are in many ways similar to me. I could take part in discussions which centered around topics of my liking, which widened my perspective. The notes shared by the participants on etherpad were very useful for future reference. I could learn a wide variety of skills including coding and Karate! Being a student, I was fully unaware of the gender issues at workplace, and Ada camp gave me an opportunity to learn about best practices for working women. The two days of the camp was fully packed up with so much of knowledge that was relevant to me.

With a fellow Ada Camper. Photo by Chit Thiri Maug

It was after the camp that the WikiWomenCollaborative, an initiative to engage women in Wikimedia, was launched. The initiative was launched by a fellow Ada Camper Sarah Stierch. Heather Walls, who designed the Collaborative’s page, was also an Ada Camper. Together, we conducted many activities including editing articles, blogging and social networking to bring more women to Wikipedia and help the existing women editors to actively contribute to Wikipedia. Meeting Sarah and Heather in person at the Ada Camp helped me overcome the cultural  and communication barriers and work collaboratively with them. It would not have been possible otherwise because of cultural and communication problems involving communicating solely online.While traveling back to India, I was deeply satisfied. I had too many projects in mind, and the potential to work towards accomplishing them – Ada Camp put me in touch with the right people and right resources to get me started. Listening to the success stories of other participants helped me overcome my initial inertia, and stimulated me to work hard towards increasing the participation of women in Wikimedia projects.

Ada Camp gave me a taste of coding. I wrote my first code in Python during my training session at Ada Camp. Though it was a small code involving adding numbers, I was so happy to have accomplished a skill! I am not good at coding yet, but the Camp helped me to get over my fear of codes. I have been improving fairly, and I dream of writing a useful code someday. Gathering ideas from the Ada Camp, I successfully conducted a conference in my city  in open space format. I have forgotten the 10 life saving karate moves I mastered during the camp, but I still cherish the learning sessions when we had a lot of fun practicing the moves on each other.

After participating in the Camp, I started spending quality time on Wikipedia on activities that are directly relevant to women. I started writing on Geek Feminism Wiki and got involved in writing blogs about women in open knowledge projects. I could get myself updated on recent issues that concern women from the Ada Camp alumni mailing list. The alumni mailing list also helped me maintain the contacts I made during the Camp. Talking at the Ada Camp increased my confidence in public speaking, and I have given three talks since the Camp.

Now, I have been involved in many open knowledge related activities that involve outreach, mobilizing people, conduct Wikipedia workshops and mobilizing funds. My participation at the Ada Camp enabled me to carry out these activities productively.

Yes, Ada Camp literally changed my life.

1. The third Ada camp is happening in San Francisco, U.S.A in June 2013. If you are a woman or an ally of women involved in open stuff, apply for the Camp here. 
2. My proposal for Wikimania includes the ideas I gathered from Ada Camp D.C. The abstract of the paper can be viewed here.

 

Ladies, let’s contribute to Wikipedia!

This post first appeared on Forbes website and Women 2.0 website in August 2012.

There is nothing else that changed my life like Wikipedia. It is not just that I “edit” Wikipedia, but I also “celebrate” it.
Editing Wikipedia is a rewarding experience as it helps me gain a new perspective of things. It is also exciting to be able to share the bit of knowledge I know with the millions of knowledge seekers from around the world.

Women and Wikipedia

Like most other tech-related organizations, Wikipedia too cannot boast of high female participation. A recent research points out that only 9% of the editors of English Wikipedia are women.
Wikipedia is sensitive to the gender gap issue and is on its way to close it. The Wikimedia Foundation has set a goal to raise the share of female contributors to 25% by 2015. A WikiWomanCamp was organized this year at Buenos Aires, Argentina for the wiki-women to interact with each other, share their experiences and discuss about gender gap and related issues.

Why so few women edit?

At the WikiWomenCamp, it was pointed out by participants that women do not contribute to Wikipedia because of various socio-cultural factors. Women are traditionally assigned the responsibility of housekeeping and childrearing, which leaves them with very little time to spend for volunteering activities. Women often suffer from lack of confidence and insecurity, which makes them think that those around them are better performers than them.
Women are not clear on why and how to get involved. The specific jargon used in Wikiprojects makes some new users uncomfortable and they eventually quit editing.
There should be a deep cultural change in the tech field to value women’s contributions and make their contributions more visible to the outer world.Sue Gardener, the Executive Director of Wikimedia Foundation says, “Deliberately focus efforts on recruiting women. Don’t assume that general outreach efforts will motivate women. Encourage women to recruit other women”.

We edit! CC-BY-SA.

 

Why should women edit Wikipedia?

Diversity of opinion is the essence of any encyclopedia. It is important that all articles are written from a neutral point of view, and having equal representation of women editors would increase neutrality and reduce bias. Women scientists, thinkers and those women who excel in traditionally male dominated fields are given lesser biographical coverage on Wikipedia than men involved in these fields.Having more women volunteers will help reduce this skewed coverage and increase the quality of the content of Wikipedia articles.

Getting involved: You too can write on Wikipedia

Anybody can edit Wikipedia. You don’t have to be an expert in the subject to be able to edit any article. You do not have to learn any computer language to be able to edit Wikipedia. If you are interested in any specific subjects, you could join the Wikiproject for the topic to collaborate with editors of similar interests and keep updated about the latest news in the subject. There are mailing lists for many aspects of wiki-editing, including the Gendergap mailing list for increasing the participation of women in Wikiprojects. Most regional languages have Wikipedias of their own, so if you are not comfortable with contributing in English, you could contribute to the language Wikipedia of your choice.
If you have photographs of educational value, you could upload them to Wikimedia Commons to permit their usage in Wikipedia articles. Wikipedia has sister projects like Wikinews, Wiktionary, Wikibooks, Wikiquote etc. which also work like Wikipedia.

For more details on how to edit Wikipedia, see this Wikipedia page or contact me!

Wikipedia: Towards closing the gender gap

This blog post was first published on Huffington Post U.K. For my Wikimania proposal with a similar title, click here.

Most of us know Wikipedia as the free online encyclopedia, written collaboratively by millions of volunteers from around the world. I am one of those writers for the last 3 years. It was by writing articles about medical sciences that I started contributing to Wikipedia. Later, I was intrigued by the enormous volume of information available on Wikipedia, and was curious to find out who actually write them – which made me delve into the editor demographics. It really made me upset when I figured out that only around 9% of the contributors to Wikipedia are women. In fact, I knew that so few women write on Wikipedia, but I hadn’t expected the figure to be as low as 9 percent.

Diversity of opinion is the essence of any encyclopedia. Having equal representation from women will bring in wider perspectives, and increase the neutrality of the articles on Wikipedia. With men creating most of the content for Wikipedia, certain subjects might be covered more than a subject that may be of interest to women. Women not writing on Wikipedia mean that certain subjects may not be receiving the attention they rightfully deserve.

CC-BY-SA. User: Jaluj. Wikimedia Commons

The Wikimedia Foundation, the not-for-profit organization that hosts Wikipedia, had recognized this problem sooner than I did. The Wikimedia Foundation has launched various programs to bridge the gender gap, and the latest addition to the list is the WikiWomen’s Collaborative.

The WikiWomen’s Collaborative was created in September 2012 by women around the world who edit Wikipedia and other Wikimedia projects, and want to encourage others to do the same. The project was aimed at helping women and transwomen to support one another and engage in programs that help the Wikimedian community to bring in new women editors. A variety of events, like edit-a-thons, interviews and wikiwomen parties were conducted as a part of the collaborative.

In May 2012, a WikiWomenCamp was organized for women in the Wikimedia movement to get together and discuss about the various issues related to being women involved in the Wikimedia community with like-minded women. Over 20 women from different countries participated in the camp to brainstorm solutions for existing problems that concern women, and suggest future plans to collaborate with one another. It is customary to conduct a WikiWomen’s luncheon at Wikimania, the global gathering of Wikimedians, exclusively for the women participants.

Organizations like the Ada Initiative, named for the world’s first programmer Countess Ada Lovelace, support women working with open knowledge projects like Wikipedia by creating resources for women in open stuff, conducting conferences and advising organizations on supporting women.

Many outreach programs were conducted in women’s universities all over the world to encourage students to participate in the Wikimedia movement. The Women’s History month edit-a-thons conducted every year, attracts both male and female editors to write biographies of notable women on Wikipedia.

The Wikimedia Foundation has set a goal to raise the share of female contributors to 25% by 2015. Given the good response from the community to various events conducted for women, it is likely that the foundation will achieve its goal within the set time limit.

If you would like to get engaged in various activities for women in Wikipedia, write to me or join us here.

Women, we can do it: A conversation with Anastasia Lvova

This is the report of my interview with the Russian Wikimedian, Anastasia Lvova.

Anastasia Lvova’s story should be an inspiration to women editing Wikipedia. She started editing Russian Wikipedia in 2007, because she found volunteering very interesting and useful to society. She has been one of the most active editors of Russian Wikipedia since. After writing her first article (certification) and improving her first good article (RFID), she became dedicated to Wikipedia.

 

Lvova. CC-BY-SA. Self.

Lvova’s contributions to Wikipedia and the Wikimedia community are impressive. She runs a bot, which does automated tasks on Wikipedia. It is now active on multiple language Wikipedias. She is also a Toolserver user — where she works on the Connectivity project — and an agent for Wikipedia’s volunteer customer service group, OTRS. She has created more than 2,200 new articles and authored some good and featured articles about Ireland and the arts. She has made as many as 404 edits in a day, 23,777 actions with flagged revision in a month and more than 60,000 edits in all! She was at the lead in organizing Wiki Loves Monuments Russia in 2011. She is an advocate for free knowledge and took part in organizing protests against internet censorship in Russia. A large part of her collection of images on Wikimedia Commons are photos from her foreign trips, because according to Russian law, photos of still-in-copyright buildings are not free.Outside the Wikimedia network, she is a photographer and writer. She graduated with a degree in management and is currently pursuing her graduate degree in psychology. She maintains a blog where she posts about her activities within and outside Wikipedia. She is also involved in charity and volunteering, and likes spending time writing letters to the elderly and children in orphanages. For her, these hobbies contribute to her activities within Wikipedia, as her hobbies help her create ideas for writing Wikipedia articles.For Lvova, being a woman editor is a positive. She says that the Russian community is receptive to woman editors, and fellow editors have helped her from time to time. She has met like-minded individuals from the community, and has done collaborative projects with them. She has noticed that the Russian wiki-community sometimes expects feminine behavior from women editors, but she says it’s not really a problem for her. She also noted that in the past, when it was hard for women to teach in universities, they became teachers, fighting against the odds, even disguising themselves as men to be able to teach. Women should be inspired by the past and feel empowered to contribute now, she argued. “Dear women, we can do it, and sharing information has always been our competence,” she said with a smile.Lvova enjoyed meeting other women editors in Argentina during the WikiWomenCamp, a meeting of women Wikimedians from around the world that took place in May 2012.“WikiWomenCamp was helpful for me not only because I got new contacts and a new perspective of things, but also because it gave me some courage to work for women’s issues,” Lvova said. She was grateful to receive a grant from Wikimedia Germany to participate in WikipWomenCamp and she has been supported by Wikimedia Poland to attend two Wikimanias and several wikiconferences.After WikiWomenCamp, Lvova started a project for new woman editors to write articles about notable women on Russian Wikipedia (they have written about 50 articles so far). She said she wishes to be helped by both men and women in her community to bridge the gender gap in Wikipedia. She thinks that this is an issue which has to be dealt with urgently. “Statistics show that around 6 to 23 percent editors are women, but we can’t be sure yet as many women prefer to disguise themselves as men because they think that a man’s opinion would be preferred over a womans,” said Lvova. She, therefore, likes to research about women’s participation in her home wiki.Her activities on Wikimedia have helped her visit interesting places, but the most rewarding experience for her has been meeting fellow Wikimedians. Through these events she has met new people who have helped her learn fresh ideas for problems, many of which were not raised in local discussions. If you want to say a ‘hi’ to Anastasia, the best place to drop by would be her talk page, where she says she would welcome the discussion.

This post was first published on Wikimedia blog on 16th October, 2012. The interview was conducted as a part of the activities of WikiWomenCollaborative.