Showing posts with label data mining. Show all posts
Showing posts with label data mining. Show all posts

Tuesday, 12 June 2012

Using AI to Combat Cyber Crime

Artificial Intelligence has, for quite some time now, been used for combating credit card fraud. Data Mining, which is in a way an application of AI, is used to detect credit card frauds by using various mechanisms. In the most general scenario, a pattern of the user's credit card usage is drawn by making use of his credit card transaction records and all the future transactions are inserted in the pattern only after conforming to this pattern itself.Whenever a transaction or a pair of transactions that violates the pattern is noticed, the system prompts the surveillance personnel to check in. Then its upon the discretion of the personnel to see if the transactions are to be investigated or to be entered in the system and inculcated in the ever changing pattern.In a more advanced form,the normal credit card usage pattern of the user is used along with a pattern of the usage seen in the credit cards of the whole group to which the user belongs. This group may be created on the basis of income,credit card category(gold,silver etc.) or even the company to which the user belongs.This scheme is more robust and more resistant against single high value transactions that may appear to drift away from the pattern but are actually genuine.

The above approaches have been quite effective in combating credit cards frauds to some extent, and as a result, agencies all over the world have started looking at AI for combating other forms of electronic/cyber crime.They sought to AI because of the fact that due to the humongous number of transactions, its utterly impossible to employ humans to track movement over the internet. They need a machine to do that and in fact they need a machine that's smart enough to match the wits of a human expert.The intelligence may either be embedded in the individual application servers, just like spam filters used by mail servers or the intelligence may be implemented at the firewalls at the gateways. The advantage with embedding it into the individual servers is that the logic related to the specific application can be embedded. E.g a traffic pattern may be acceptable if destined for mail server but not for some office application server.In fact the best approach is to divide the intelligence amongst the two places. General intelligence is embedded at the firewalls and the application specific intelligence is embedded in the individual servers.

The general model suggests that some traffic analysis technique be used. This technique would differ according to the networks. Traffic could be analyzed at one or all levels. Either only the datagarm traffic could be analyzed or the ip level traffic or both. The traffic is again matched with the general pattern of traffic just like pattern matching in credit card fraud detection. At the firewalls, the overall traffic pattern is analyzed, and at the individual servers, the application level and session level traffic is analyzed. At the application level, once again two patterns could be used - a user pattern and a group pattern. At the firewalls however, a single pattern has to be used.In fact, the system may keep different patterns for different days or different times instead of a single pattern, and these different patterns may then be used accordingly.Like every cognitive learning mechanism, these patterns would also improve with time. The system would match actual pattern with the stored pattern and also keep changing according to the patterns that it analyzes. For example, if the system reported an anomaly and the network admin thinks its normal traffic, the system would inculcate this in the traffic pattern model and would improve itself. Hence, with time, the system will become more and more effective. 

Thursday, 10 November 2011

Predicting Earthquakes In Advance

Surprised after reading the title of this post ? More than that, you might be wondering if it is possible. Well, this may turn out to be a reality in the near future.Earthquakes, are perhaps one of the most devastating forces of the nature. Earthquakes, ever since the inception of civilizations, have claimed countless lives and have resulted in heavy damage to property. Whereas, damage to property can be controlled by making earthquake-resistant structures, loss of life can be ensured by both, making stronger structures and finding ways of predicting the earthquakes, well in advance.

Now, the questions is, how ? Earthquakes are of two types - shallow ones and deep ones. Shallow earthquakes are the ones that originate within a depth of around 300 KM beneath the surface of the planet, and Deep earthquakes are the ones that originate at larger depths. The reason behind the shallow earthquakes is very well understood, however, there is no clear-cut explanation for the Deep Earthquakes. Hence the concept that will be used to predict earthquakes is applicable only in the case of shallow earthquakes. Moreover, shallow earthquakes cause more loss and wreaken more havoc than the deep ones.







Now, the principal point is that, the shallow earthquakes have got a definite relation with seismic activity and seismic waves.They are basically the waves which originate because of the movements inside the Earth.The seismograph shows more activity in case of actual earthquake and the Richter Scale measurement of Earthquake magnitudes, is actually the magnitude of largest variation on seismograph recorded during the Earthquake's span.The monitoring centers throughout the globe, keep recording the seismograph of corresponding zones, and this seismograph easily tells when we had an earthquake.The monitoring centers are placed after analysis of tectonics(There are several tectonic plates inside the earth and the shallow earthquakes are related to the collision and other interactions amongst the tectonic plates. This is tectonics).It also gives us information about the various parameters regarding the geographical area to which they pertain. Like, you must have heard about the danger zones, in terms of probability of occurrence of an earthquake. Countries and states are divided into seismic zones.Some zones have a high risk of seeing an earthquake than others and some zones are also likely to see more powerful earthquake than others. This zonal distribution turns out to be very useful during planning . Such zones are made after analyzing the seismic activity over a long period and also after analyzing the tectonics of that place. Like places that are closer to the meeting point of two tectonic plates are at a higher risk .

So, the question is, can't we make better use of the seismographs and use them for better things than just planning zones? Well, seismographs may turn out to be the biggest boon for mankind Seismographs are formulated by measuring the strength of seismic waves and they are analyzed across various parameters.Seismographs are recorded at all times, and most of the places in the world will have a large database of seismographs by now. Now, there are two suggestions for making use of these seismographs in predicting future earthquakes - A statistic-based philosophy and a Data Mining based philosophy. The statistic-based philosophy is a conventional one.The seismographs of all the years till now are analyzed and the values of various parameters are calculated. The values during the earthquakes are given higher weights while compiling a area-based formula that can be used for predicting earthquakes. The current values of the formula's application help us in finding out if we are nearing an earthquake. Now, the disadvantages of this approach are :

1. values in future may be similar to the past values just by chance and hence may turn out to be false predictors, in the end.

2. the approach might predict an upcoming earthquake, but it will not predict it well in advance and the authorities may not get the required time for letting people know of the same.

3. every statistic approach has its own disadvantages.

4. if there were heavy variations in parameter values during earthquakes, then the formula for that area would be very fragile.

5. The process used for computation of the formula is based on knowledge of a subject that is not well understood. So, the approach is not perfect.

The second approach though, is the one that should interest us the most. It is based on Data Mining. Data Mining is basically a phenomenon, in which tonnes and tonnes of existing data is analyzed by a data mining program and the attempt is to find out some hidden and potentially important information. This information may be in terms of hidden relationships between different items or may be anything else that holds a lot of value for the organization to which the data belongs.Data mining can only be done when you have tonnes and tonnes of data to mine. Just to give you an example, consider a Departmental store. The departmental store sees tonnes of visitors everyday and all information regarding all the billing gets stored in their databases. When the database grows large enough, it is combined with the even older databases and all the billing information stored till date is moved to a Data Warehouse(it is just like the Data Archive of a organization). Now, the departmental store wanted to find out any hidden information from this archives(since the archive is humongous, manual mining is not an option).They run a data mining tool on this data and they find out that about 50 percent of the users who brought bread of A brand also brought cheese of B brand. This is a very value information in this context. The store may give an offer, where a combo of A brand bread and B brand cheese is given. Now since, 50 percent of the users were already loving this combination, a great percent of those who haven't tried it yet, will also have an urge to try the new combo. The store can reap in huge profits like this. That is Data Mining for you.

So, in context of Earthquakes, what has this Data Mining got to offer us? Well, it can do wonders. The thing which has long been recognized by seismographic experts, is that seismographs of most areas might show some specific behavior just before the earthquakes.Now, we don't know how long this behavior lasts or what sort of behavior it is. But we do know one thing.We have the seismic activity recorded, both in terms of graphs and also in terms of values. And we can also assume that we have a significantly large seismograph database . Now, by nature, seismographs are going to give you a lot of data. Seismographs are continuously recorded all the time.We have a sufficiently large  number of data mining tools for both, mining graphical data and mining numerical data. Hence,if there is any behavior, Data Mining will find it and tell it. In terms of graphical data mining, the tool may come up with some pattern that was experienced some time before the earthquakes or through some time before the earthquakes, and in terms of numerical mining, the tool may come up with a set of values that was seen some time before the earthquake, or may even come up with some averages.Hence, if we do have some pattern in the seismographs and we have appropriate seismographic data for some area, then an effective data mining process will always come up with this hidden behavior and experts can use this information to formulate models.In fact, Data Mining tools, also provide the lower level details behind its findings and help the experts in making detailed models.A different program may monitor seismic activity against this model and report results to experts at all the times. Now,it does not matter that whether the behavior was transient or it was prolonged.If there was a specific behavior, Data Mining will find it.The strength of Data Mining lies in the Artificial Intelligence that the various tools possess. Data Mining tools use neural networks, genetic algorithms, cluster algorithms and various other approaches to analyze the data across various dimensions and come up with hidden information.But,this approach too has a few drawbacks :

1. The behavior may not be very useful if that was exhibited just a few seconds before the earthquake.

2. The Data Mining tools take a lot of time for mining information, hence using the tool on the go, is not possible. One has to properly plan that when the latest Data Mining session has to be run and after collecting how much new data, should it be run.

3. Data Mining, at times, may come up with a lot of possible alternatives for explaining a particular piece of information. This is not the fault of the tool, this is a because of the nature of the case. In this case, experts will have to use their knowledge to reduce the number of cases to formulate the final model.

So, the best thing that we can do, is to combine the first approach with the second approach and make a combined model that can be used for predicting earthquakes. There's no doubt that a lot of capital and time will be spent, but just imagine the benefit it has for the mankind.Some research has already started in this field. A team from Indian Institute Of Technology(IIT),Hyderabad is working on a project, where several small sensors will be placed in the Himalayan belt and Data Mining will be done to predict earthquakes, a day in advance. The sensors are from Japan, so their teams too are a part of this. Teams from other IITs will also be contributing. The project will get into full flow by 2015.Some more research from other universities throughout the world, is under way. We can just hope that this research comes up with some encouraging results and gives us a model by using which, areas from all over the world, can find out if an Earthquake was approaching, and that too, well in advance. Just imagine the world then. That is what technology can do.