Upcoming talk

On my upcoming talk at QMDNS (Linky)...

Title: Modeling and Detection of Sudden Spurts in the Activity Profile of Terrorist Groups

Abstract: The main focus of this work is to detect sudden spurts in the activity profile of a terrorist group and track it over a period of time. Towards this goal, a $d$-state hidden Markov model (HMM) that captures the strength of the group and thus its activity profile is developed. The simplest setting of $d = 2$ corresponds to the case where the strength is coarsely quantized as Active and Inactive, respectively. Two strategies for spurt detection and tracking are developed here: a model-independent strategy that uses the exponential weighted moving-average (EWMA) filter to track the strength of the group as measured by the number of attacks perpetrated by it, and a state estimation strategy that exploits the underlying HMM structure. The EWMA strategy is robust to modeling uncertainties and errors, and tracks persistent changes (changes that last for a sufficiently long duration) in the strength of the group. On the other hand, the state estimation strategy tracks even non-persistent changes that last only for a short duration at the cost of learning the underlying model. Case studies with real terrorism data from open-source databases are provided to illustrate the performance of the two strategies.

Short Summary: Existing work on activity profile modeling fall under three categories: i) classical time-series techniques of fitting trend, seasonality and stationary components, ii) a technique that goes by the name group-based trajectory analysis (derived primarily from statistical methods in sociology research) where terrorism trends with similar developmental paths are identified, and iii) self-exciting point process models. The first method has been around from the early 80s in different forms, the second method corresponds to early 2000s when many open-source terrorism databases came into the picture, while the third method is rather recent. Of these, the third method is the most exciting (pardon the pun!) as it allows the full-blown hammer of spatio-temporal point process techniques combined with statistical methodologies in hypothesis testing, inferencing, parametric and non-parametric methods, etc. In some sense, this modeling method has been motivated by similar trends in gang warfare, seismology, epidemiology, etc., where self-exciting models have really pushed the frontiers of statistical inferencing. In terms of negatives, the model has too many parameters that have to be learned for all these inferencing techniques to work nicely.

My work is motivated by two central issues. One, terrorism (despite the huge media attention) is a rare phenomenon with data-collection a painful manual exercise (despite the rise of many open-source databases). Thus, parsimonious models driven by simplistic hypotheses make really good sense in this business. Two, while self-exciting models make perfect sense for gang warfare where revenge attacks can be nicely modeled as self-excited and in seismology where aftershocks are driven by primary shocks, it is not clear if this is the best framework to be used for terrorism modeling. In this context, I propose an alternate HMM framework where the hidden states model certain qualitative attributes of the terrorist group which could undergo a shift over time and thus drive the activity profile in different ways. I show via statistical techniques that this model is a good competitor for self-exciting models and has the added advantages of being described by few parameters (four, in the simplest non-trivial setting), and being motivated by a simpler set of hypotheses on the bare basics of how terrorism works. To rephrase Peter Diggle, at the end of the day, the objectives of a statistical analysis should be determined by the objectives in collecting a certain dataset under question. To address this issue, I show that the proposed approach detects and tracks spurts in the activity profile that can be easily identified with certain geopolitical realities. This is done with data on FARC, Shining Path and the Indian Maoists.

Two education related statistics (Pakistan, India and China)

On adult literacy levels in Pakistan over the years, from the article by Anne Goujon and Asif Wazir of the International Institute of Applied Systems Analysis:
Year -- No education -- Primary Incomplete -- Primary Complete -- Lower Secondary -- Upper Secondary -- Higher Education Qualified (first Male, then Female -- all numbers in percentages)

1972 M 68 03 10 14 03 02
1972 F 92 01 03 03 01 01

1981 M 66 -- 12 16 03 03
1981 F 88 -- 05 05 01 01

1998 M 48 06 12 24 05 06
1998 F 74 04 07 10 03 03

2005 M 34 05 15 29 08 09
2005 F 64 03 11 13 04 05

Observations:
1) From the above data, the percentage of literates in Pakistan is growing (in both males and females) over the years -- which should not be very surprising. Every country makes progress on literacy levels if you give them enough time.
2) But this is not how to read the above statistics. To me, the differential in literacy between males and females is a clear indicator of the level of violence in that society (see related posts: Linky 1 and Linky 2). In Pakistan, the illiteracy differential between females and males in Pakistan is 24% in 1972, 22% in 1981, 26% in 1998 and 30% in 2005.
3) From their more detailed paper (Linky), the literacy differential between males and females is 7% in 1951, 19% in 1961, 8% in 1972, 19% in 1981, 23% in 1998, 24% in 2004 and 24% in 2009. On both extremes (illiteracy differential as well as literacy differential), Pakistan is in pakistan (aka deep do-do) and the level of violence is consequently very much explainable.
4) Breaking these figures into provincial statistics from p. 13 of Linky, we have the following differential literacy rates:

Year Punjab Sindh NWFP Baluchistan
1972 19% 19.9% 18.4% 10.5%
1981 20% 18.1% 19.3% 10.9%
1998 22.1% 19.72% 32.57% 21.5%
2004 21.3% 19.80% 38.15% 26.41%

5) Clearly, Punjab and Sindh have stabilized in terms of differential literacy rates and if my hypothesis is correct, then the violence levels in Punjab and Sindh should be pretty much consistent from 1970 and up. Either this trend was not recognized before or the hypothesis of literacy differentials correlating to violence is not completely capturing the picture or the data is being fudged somewhere.
6) On the other hand, NWFP and Baluchistan see a massive jump in the differential literacy rates in the post-Soviet/Taliban setting. While the revolution in Iran (1979) marks the turning point in terms of an upshot for worldwide terrorism trends, the violence level in the neighborhood is slow by a good 10 years. This is not explainable. While terrorism/revolutionary activities spills over very quickly into the neighborhood, why is there a lag in Pakistan? Was it that the CIA arming of mujahideen also come with caveats that prevented a quick Talibanization of the Pakistani mindset?

Salary for higher ed in India, following up from Linky. Some comments from Linky:
1) Two countries -- China and India -- have been the focus of many global education watchers in recent years as they have moved rapidly to expand capacity and expertise in their university systems. The study shows India holding its own in international faculty salary comparisons (factoring in cost of living), but not China. This reality has led Chinese universities, Altbach noted, to offer very high Western-style salaries, to a very small number of academics (typically Chinese expats recruited home). The numbers are such a small share of the total Chinese academic labor pool that they don't influence the Chinese totals, he said, but without these deviations from salary norms, China couldn't attract those researchers. India, in contrast, does not permit universities to deviate from salary norms for superstars.
2) Another area where the countries differ is in the difference between entry-level salaries (averages for assistant professors) and those at the top of their fields (full professors). Across all 28 countries studied, the average ratio of the senior salary average to the junior salary average was 2.06 to 1 (factoring in the PPP). The gaps between senior and junior pay levels were greatest in China (4.3 to 1) and smallest in Norway (1.3 to 1). Western European nations generally had low ratios.
3) Monthly Average Salaries of Public Higher Education Faculty, Using U.S. PPP Dollars

Country -- Entry Level Pay -- Average Pay -- Top Pay

Armenia $405 $538 $665
Russia 433 617 910
China 259 720 1,107
Ethiopia 864 1,207 1,580
Kazakhstan 1,037 1,553 2,304
Latvia 1,087 1,785 2,654
Mexico 1,336 1,941 2,730
Czech Republic 1,655 2,495 3,967
Turkey 2,173 2,597 3,898
Colombia 1,965 2,702 4,058
Brazil 1,858 3,179 4,550
Japan 2,897 3,473 4,604
France 1,973 3,484 4,775
Argentina 3,151 3,755 4,385
Malaysia 2,824 4,628 7,864
Nigeria 2,758 4,629 6,229
Israel 3,525 4,747 6,377
Norway 4,491 4,940 5,847
Germany 4,885 5,141 6,383
Netherlands 3,472 5,313 7,123
Australia 3,930 5,713 7,499
United Kingdom 4,077 5,943 8,369
Saudi Arabia 3,457 6,002 8,524
United States 4,950 6,054 7,358
India 3,954 6,070 7,433
South Africa 3,927 6,531 9,330
Italy 3,525 6,955 9,118
Canada 5,733 7,196 9,485

Rejoice, for ye hockey has won -- April 2, 2012

As the World Series Hockey has winded down, some notes from the event:

1) The tournament as a whole was dud, even if the commentators were noting it for being the epitome of brilliance. The primary reason for this dud-ness is that many of the famous/mainstream Indian hockey stars did not take part in the event. What we essentially saw at WSH were the "discards", the "fallouts", the "also-rans" and the "tomorrows". In terms of foreign stars, we saw the fading Imran Warsi (who could still share the Golden Stick at the end of the event), Rehan Butt, Shakeel Abbasi, the retired Brent Livermore and the almost-retired Ken Pereira. Why else will anyone fight it out with their own hockey federations' in a see-saw battle with the FIH? It is a singular matter of pain for the hockey fan to not see this country's very best in action,
2) for which, the complete and unanimous responsibility lies with Hockey India. The games, the machinations, the insinuations and behind-the-scenes actions left those who care with no doubt as to how arbitrary its actions can be/are. It also reinforced my beliefs that hockey players care only about an Olympic medal, not even a World Cup win matters more. There are many historical/legacy reasons for this first among equals syndrome, not to mention the etched-in-gold tale of Indian hockey at the Olympics.
3) There was enough money in the kitty and some of the overnight richie riches (duly deserved, of course!) are going to be plain confused and retiring, or more striven to win more. There are no middle grounds in how this money will impact the players. For the record, this is the biggest prize money event in the hockey world, let alone in India. And Indian players sitting at home due to the fiats of Hockey India are not going to remain silent in the post-Olympic scenario where the world comes to an end not till 2016. And if India does win a hockey medal (on which, I am willing to take some early bets!), the players shall rule the popular/public's attention span for far longer than the World Cup winning Indian cricket team did.
4) Either Hockey India creates its own version of WSH and set the ball rolling before the 2013 version of WSH, or Hockey India is going to implode due to the public furore of suspensions, resignations, walkouts and dropouts of the mainstream Indian team. In one swift move, Hockey India faces a do-or-die battle that even Shri. Ajay Maken's best efforts could not resolve. This was essentially the same calamity that BCCI faced when the infamous ICL ruled the early roost over IPL. But the BCCI administrators are sensible businessmen with a nose for money and common sense. Unfortunately, Hockey India is known more for its riff-raff than common sense or administration skills (a phrase that goes for far longer in the hockey community as the 1964 Gold does, to be charitable!).
5) That said, by its own self-contradictory definition, WSH cannot sustain itself without some serious ways in which the mainstream Indian talent can be co-opted into its fold. Either it disbands its union with the IHF and seeks a truce with HI and the FIH (for which Bridgestone has to talk with FIH with money on the table), or WSH renders HI and FIH pointless by a public furore and stings which are easily plantable (like the one that was forced upon the nefarious and thankfully forgotten K. Jyothikumaran).

6) That brings us to the stars of the event: Gurjinder Singh (Linky), all of 18 years and a future Sandeep Singh in the making can be safely said to be the find of the tournament. Other finds of the tournament include Vikramjeet Singh (24 years - Linky), and Gurpreet Singh (26 years - Linky). Among veterans, Prabhjot Singh (31 years) who has not been in the Indian hockey matrix since the days of Jose Brasa, the fallout in Canada and the middle-finger gesture in the World Cup and Deepak Thakur (31 years) did pretty well to have folks get behind them in calling for Mike Nobbs' attention. Roshan Minz (25 years - Linky) and Adam Sinclair (28 years), though not old by any means, are veterans by stature and also get to be in the top 48 to have a chance to reclaim their place in the top 15/18. Maninder Singh Wallia has a take on other stars of the event at Linky. While Len Aiyappa had a great event, he is not in contention for the Indian team and that makes it a sad tale of talent lost prematurely in the bygone era under the sacred tutelage of Hockey India and its precursor, the Indian Hockey Federation. It makes some coaches and administrators criminals and some of them less wiser in hindsight than otherwise.
7) It is a typical commentary on the drag-flick and its impact on the world game that three of the coming stars (Gurpreet, Gurjinder and Vikramjeet) are drag-flickers, who got inspired by the stable of Baljit Singh Dhillon, Jugraj Singh and Sandeep Singh. In terms of defence, which we sorely need and lack a certain depth of, contenders include Ajmer Singh and Damandeep Singh.
8) Diwakar Ram was ruled out before the event due to injury and will be ruing another opportunity gone a-begging. Rajpal Singh and Arjun Halappa would like to call the event a flop given their high standards. Karan Bhaskaran (son of Coach Vasudevan, Linky) and Devender Walmiki faded out in the second half. Comeback star Gagan Ajit Singh must have felt that the game has changed since he left a few years back.

9) At the end of the event, Coach Rajinder Singh was heard saying, "I am still one of the very best!" And rightly so. Former junior coach Harendra Singh and Jude Felix (who interviewed for the spot that Mike Nobbs now holds) will also feel a need to be happy. In a jiffy, we now have four back-up possibilities including the Pakistan bound Ramandeep Singh. The sour note must indeed be felt by Mukesh Kumar and the completely forgotten Shri. M.K. Kaushik. The pool of quality umpires that India has is definitely bound to increase and the story of not sending enough technical delegates/umpires to world events could be a past.
10) Coach Mike Nobbs will have to seriously consider some backroom diplomacy that can bring the "fallouts" into the Indian picture again. The time-out on Prabhjot Singh has to end for Prabhjot is too big a name to be kept out for too long. Deepak Thakur, Adam Sinclair, Roshan Minz -- the names that need reconsideration are long as pointed out earlier. As Coach, Mike Nobbs rules the roost and while noone expects him to do a Greg Chappel, noone expects him to do a Gerard Rach (if you remember him!) either.
11) The last semi-finalist was not spotted till the last match of the league phase as the Imran Warsi led Chennai Cheetahs caught a Klusener bug. While on the one hand, that could mean that four awful teams were not around, a look at the Points Table also shows that Chandigarh Comets and Sher-e-Punjab are a cut above the rest (even if Chandigarh did nt enter the finals). In other words, there is first Punjab in hockey and then the rest. Meanwhile, the hockey center of Karnataka, Orissa and increasingly Bombay and the Northeast will debate this statement endlessly and I am not too unhappy seeing a contender rise as that means more depth in the charts, which we sorely need. In terms of hockey academies, Sansarpur had one famous contender whereas Panposh had one other star contender in the mix (to my best checking), the rest came from all over the place. This is not alarming, nor is it a joyous occasion, but just a data-point that needs further observation.
12) We saw eight well-maintained stadia in different corners of the country and we saw more people streaming in towards the end phase of the event. Whether these stadia will be kept in such great condition or whether these fans will return in the future are questions for which noone has a clear answer. While, in general, such philsophical questions have no unambiguous answers, the case of Indian media/public/administrators' attention to these matters is abysmal.
13) And the WSH taught a lesson or two to Hockey India and Indian Hockey Federation in running a professional webpage with social media connectivity. It is a crime to claim to be the official national sports federation and not have a ready list of entries on hockey in India for which both HI and IHF have to take a rap on the knuckles.