CSM meeting minutes 2.003

Meeting Minutes

Sunday 14th December 2008

Present

Ankhesentepemkah, Bunyip, Darius JOHNSON, Extreme, Issler Dainze, LaVista Vista, Meissa Anunthiel, Omber Zombie, Pattern Clarc, Scagga Laebetrovo, Sophie Daigneau, Tusko Hopkins, Vuk Lau, CCP Arkanon, CCP Chronotis, CCP Diagoras, CCP Dr.Eyejog, CCP Falout, CCP Wrangler, CCP Xhagen, GM Grimmi

---

Announcements / Elections

none

---

POS Exploit

Rumours on the Scrapheap Challange forums claim that certain people had been using this exploit for 4 years. How long has this loophole been exploited?

Eyejog was able to confirm exploits starting in March 2009, with the bulk of it becoming operation in May/June 2008. CCP is in the process of restoring older data and will mine that data over the next two weeks.
Meissa asked how far the current database extends. Eyjo responded that different data sets go back to different time periods, and as an example they know the exploit existed earlier than March, but they haven’t been able to find anyone using the exploit earlier than that yet. However they need to restore the older data so that they can mine it to see if people were using it earlier and stopped.
Eyejog would like to know if CSM thought it likely people were using this for a while and then stopped.
Pattern asked if this exploit corresponded to a new UI or another addition, and did the current data allow them to see that far.
Tusko asked if it was theoretically possible for some to be using this exploit for 4 years. Have CCP checked to see if the bug was inserted when POS’s were introduced?
Bunyip asked if the exploit was tied in with Alchemy in any way? Eyejog said no, simply a coincidence.
Diagoras confirmed the exploit as being possible from at least February 2007, however they are still investigating further to see exactly when it became possible.
Issler is interested to find out if a petition was opened (as claimed) 4 years ago. Eyejog noted this would be answered later in the meeting.

Why was this issue not detected sooner if the impact had "considerable and far reaching" consequences? After all, if items were generated out of nothing in sufficient quantities to have an effect on the economy, some alarm bells should have rung at some point. If this issue was not reported, how long might it have continued for?

Eyejog noted that the reason it wasn’t detected sooner was that the exploit wasn’t started ona grand scale until this year, so the effect on the economy came into being gradually. From a QA perspective, they test for potential exploits regularly that are perceived as risks both on TQ and before they release new features or functionality, but in a universe as complex as eve, it’s not possible to test for all potential exploits and in this particular situation they didn’t discover the issue earlier. QA has been working for the las few days to reproduce and assist in solving the exploit.
Issler pointed out that this was a strange bug to have been introduced this recently since there weren’t any changes with POS reactions since the timespan noted in the answers to the previous questions.
Diagoras explained that February 2007 was the earliest date they had that they are able to check the code for the exploit that hadn’t been changed in a way that would effect this, they are still working on verifying exactly how long it has been possible.
Vuk noted that every big patch changes a lot of unrelated things, so the bug could have been introduced at any time.
Bunyip wanted to know how much Alchemy would make up for this epxploit, will the values equalize or will dyspro-based prices increase. Isller noted that minerals used for alchemy are pretty rare.
Tusko wanted to know if the exploit didn’t cause obvious anomalies in the data, how did CCP conclude it had considerable and far-reaching consequences? Was it an early guess?
Diagoras noted that the volume of materials produced vs. the normal possible supply of material into the game via moon mining.
Eyejog re-iterated that to CCP it looks like the bug was available for a lot longer than it was used on a grand scale by players, but we should remember that this is still something they need to verify through datamining.
Eyejog also commented that in regard to the economic scale, when something enters into the system slowly, but then increases at a faster rate, it does not show up as an anomaly in the data, but they had 2 hints which they are investigating – the first being unusual price movements in Morphite in July and the second was the price decline of Ferrogel in recent months.
Darius JOHNSON noted that CCP said that the activated wouldn’t be an anomaly in the data, however this exploit amunts to a dupe, resources were being introduced into the system that were never mined or created, that would then lead to a situation where potentially you have more of the resource being sold than is possible. Given that the situation was said to be far reaching, shouldn’t it have been something that was noticed specifically, and if it wasn’t an audit point, would it be in the future?
Eyejog agreed and explained that the problem was that these materials were used directly for production or traded outside of the market, so in order to see it, they would have to be looking for it specifically. They will be reviewing their audit processes to see if they can create automated audits that give them an indication of mismatch in resource allocation.

Adding to the above, are there any plans to add systems to detect future exploits like these?

Eyejog noted that they are reviewing the processes based on this both with regard to QA and market monitoring and data mining.
Darius asked if the introduction of Alchemy will make this more difficult, but noted it wasn’t really relevant to this discussion.

How much contraband resources were produced through this exploit, and how big a share of the overall production is this?

Eyejog stated that the exact numbers weren’t known currently, but will be part of the entire investigation. Diagoras provided the rough estimates: Roughly 35% of the Ferrogel market. He noted that that is a very rough and an early figure which needs further analysis.
Meissa asked if they were only monitoring Ferrogel, or have the seen use of this exploit for other reactions? Diagoras responded that the primary materials being produced were Dysporite, Fermionic Condensates, Ferrofluid, Ferrogel and Prometium. Ferrogel was by far the most exploited material.
Pattern asked if they could put that number into isk terms or into the likely increase or effect on the market that would now ensue after the bug’s correction. Diagoras stated that they can’t give isk numbers currently as it required further data mining.
Lavista wondered if it is even something CCP can talk about, as even if they could answer it, it would be a bannable offense. Eg. “prices will go up x%” and person Y foes out and buys up stock. Eyjog noted that CCP have already said that they expect this to impact the
market, and data in Jita shows the impact already. However, they plan to let the market settle a little more before they give out estimates as there is a lot of speculation occurring currently and they need to allow that to settle.

In the light of the above, what is the scope of the impact this exploit had on the economy, and what will happen to the economy now that the loophole has been closed?

Eyejog split this into 2 parts: The impact on the economy is significant; overall they are expecting this generated a few trillion isk. As a comparison, the daily trade in eve through the market is 3 trillions isk. So it’s significant, but not catastrophic.
Part 2: Due to the size of the Eve economy, Eyejog is expecting the market to recover quite quickly, the introduction of Alchemy will also help in that regard. However the most critical materials they are monitoring closely and will take further action of needed so that there will be no absolute shortage of the material in question.
Issler asked if these expectations are assuming that the dates currently thought to be the first major use of the exploit? Eyejog answered yes.
Bunyip wanted to know what kind of impact the exploit has had on the ETC market, and whether the effects will ever be fully recognized? Eyjog answered that it wasn’t known yet.
What impact did the ISK and resources made through this exploit have on the outcomes of 0.0 warfare?
Eyejog said that it wasn’t known at this point and most likely will not be possible to estimate.
Meissa asked if it was possible to determine what the resources gained were used for at all – eg. ETC trading, POS’s, capital ships? Or was most of the isk sitting in wallets?
The answer to this was restricted by NDA, and further discussion will be had with CSM in the face-to-face meeting in January.

If there is serious shortage on certain materials now that the exploit has been fixed, are there plans to make them (slightly) more common?

Eyejog answered that no, the only plan for now is to monitor the situation and make sure there will not be a shortage of goods. Shortage equals absolute shortage, i.e. product not available.
Meissa asked how they planned to go about that? There would be less of the high-end moon product, will CCP act as purveyors of those goods? Eyejog replied that if there was an absolute shortage, then CCP would use NPC market orders, but he would like to emphasize that they are not expecting that they will need to do that.

CCP decided not to make public the names of the corporations and individuals that were involved in this exploit. This lead to wild speculation and accusations, but after a day the people found out who most of the people involved are anyway. Would it have been a better idea to make the names public from the start?

Arkanon stated that CCP’s policy in regards to this has been consistent from launch; they are sticking with that policy. Changing that policy on a whim is seldom a good idea.
LaVista asked if CCP could provide n updated number – was it still 70 accounts, 3 corps and 2 alliances? Grimmi replied that they are the rough numbers of bans issued, but investigations are still underway.
Darius asked if CCP had considered the feasibility of making more public announcements of bannings for things such as exploits and RMT – the way wow handles it as an educational deterrent. He assumed bannings occur, but it’s rare that an announcement or news item of any type is made. Arkanon stated that they do announce bannings and actions taken when it’s of a noticeable scale, but naming names is a breach of CCP policy on player privacy. Reporting on day to day activities of the GM team is not something CCP feels would be very effective as the bans are not newsworthy most of the time.
Vuk agreed that exploiting of this scale should be publicly announced, especially when there are rumors about all major alliances being involved and previous history of trolling, not to mention that it possibly affected alliance warfare.
Darius wanted to note that he was in no way insinuating that names should be named, but rather that if regular updates would be given regarding numbers of bans it could be effective as well. Possibly a monthly report of some type exploring the avenues that better communication from the GM team regarding things like actions taken, response times, tickets opened, etc. would create. Pattern agreed. Arkanon thanked Darius for the suggestion and while he wouldn’t speak on behalf of the GM team, he can say that their approach is deliberate. Whether it is the correct approach is debatable.
Issler stated that he didn’t understand how naming a character that was banned has anything to do with privacy, as there is no published link between a character and the RL person affected. Arkanon restated that they draw the line at naming names. Eyjog pointed out the current CCP policy is clear, but the CSM should file this as an issue to discuss in the Iceland meeting.

What guarantees do we have that everyone involved has action undertaken against his/her account? Did CCP track down the people that did not directly own one of the offending POSes, but was involved in the process regardless? What about the people that had set up POSes to exploit this issue in the past, but removed them or lost them through combat before CCP discovered the exploit? On the other hand, what guarantees do we have that there were no innocent people punished?

Eyejog stated that CCP is currently working on this issue, defining who is and is not guilty is quite tricky. Ie. If someone received battleship from his corporation to participate in a fleet battle, is that person guilty of an exploit? His question to the CSM is , how far should CCP go? Is it possible to use fines instead of bans for those enjoying the benefits without being directly guilty?
Lavista noted that guilty by association is wrong, the people that ran the operation should be banned. Things like titan’s that were funded by indirect isk should not be banned and from the GM’s POV it would be hard as at some point everyone would be guilty.
Issler noted that if he had a battleship funded by this it should be removed, and it would make people think about the people they choose to associate with.
GM Grimi pointed out that as in any investigation, they need to establish with beyond a reasonable doubt that someone was directly involved in the exploit or knowingly gained from it in order to take punitive action against them. Merely being in a corporation or alliance does not necessarily get people in trouble. Scagga agreed with Grimmi.
Vuk thought that it shouldn’t go too far when it comes to banning as there would be more cons than pros, as shown with the T20 incident. Members shouldn’t be considered guilty if they weren’t aware of the exploit, banning and fining should be limited only to the people directly involved in the exploit.
Eyjo noted that fines have not been used in eve before, they have seized illegal isk but putting an actual fine on an offense would be something new – what did the CSM think of that? Vuk stated fines are pointless. Issler is happy for resources or value of resources are removed from wherever it ended up in cases of no direct involvement. Scagga is opposed to fines as they are meaningless and seizing illegal isk/assets is a good limit. Vuk questioned CCP’s double standards when it comes to banning previously. Arkanon asked vuk to elaborate. Vuk stated that there were cases of people being banned on circumstantial evidence when dealing with RMT. Arkanon responded by stating that anyone who felt they were wrongly banned can petition to argue the case.
Xhagen reminded everyone to stay on topic – bans in general are not discussed, this is a discussion regarding the exploit committed. Vuk stated that everyone knew that the petition system is far from perfect, and in most cases the answers from low level GM’s are generated.

Is it true that this issue has been reported through the petition system and through email three years ago? There are forum posts from that date that indicate problems with the starbase reactions, and the people that reported them claim their petitions and emails were not answered.

Eyejog stated that they were still trying to verify this. The petition system has changed and CCP had outside contractors for customer support at the time. It will take time to look through old databases and see how the old system worked.
Scagga wanted to know if it was possible for GM/Customer Support to delete petitions from the system; essentially cover their tracks. Xhagen noted that as an ex Senior GM at the time, the answer is no. Once a petition was filed, it could not be deleted by a GM of any level. Scagga asked about emails that may have been sent about this issue. Xhagen answered that the GM’s had an outsourced email system prior to 2005, and that it will take a while to dig through, but the same applied to emails, once a ticket was xreated it could not be deleted. Post 2005 all petitions are available.
Issler asked if he ran into a similar situation in the future, petitioned it, got a generic response that no issue was identified, could he then safely do whatever it was he found?
Eyejog responded that no, you should know the TOS and EULA and any breach of it means that actions would be taken against the offending account when the issue was discovered.
Darius pointed out that it would be stupid to continue to cheat simply because your petition wasn’t answered. Issler responded that it meant that he would have to decide on his own whether something was a defect or an exploit. Eyejog stated that in short, a petition shouldn’t be handled this way in the future, so if people find themselves in this situation and not being answered, refile the petition, if not answered then, contact Internal Affairs.

CCP say they are improving the petition system to deal with petitions that report exploits faster. Could we get any more details on how they plan to do this? And what guarantee do we have that exploit and bug reports do not get a standard GM reply and closed?

Eyjog stated that generally speaking, petitions filed in the exploits category are immediately assigned high priority and get attended to within the day. Sometimes however (like this case) petitions are filed wrongly risking a delay in response. CCP have already set up new work procedures that will discover wrongly filed petitions earlier, but the exact details are still being finalized. Another option discussed was to create a tool similar to the fleet fight notification system.
Vuk wanted to add quickly that the current petition system was not good, mostly due ot the incompetence of the low level GM’s and nothing is solved until escalated to senior GM’s. Vuk also thinks that CCP’s multiple petition filing system is wrong. Eyejog thinks this should be something added to the Iceland meeting agenda.
Grimmi expanded on Eyejog’s statement in that CCP are working on inventing means of detecting and re-categorizing misfiled petitions in order to make sure this type of thing doesn’t go unnoticed. No specific details at this point however.
Were any CCP employees or GM staff involved in this the abuse of exploit in any way, either directly or indirectly (by keeping it quiet or by informing players, for example)?
Arkanon answered no, there is no indication that a CCP employee was involved in this at all, but investigations are continuing as more data emerges.
Scagga noted that he had been informed and read from various sources that people came across odd anomalies or said that they knew of instances where GM’s were rumoured to be involved; where would the investigation draw the line between following a wild goose chase and following a lead.
Arkanon replied that people should feel free to email them with any ideas or leads you may have (internalaffairs@ccpgames.com), they will look into it.
Eyejog wantedto reassure everyone that CCP are leaving no stone unturned and that Internal Affairs agents are very quick to spot if certain leads look promising or wild goose chases. So if people think they have a lead, please let IA know.