tag:blogger.com,1999:blog-33713133752517954692024-03-08T01:57:58.702-08:00all about cspCSPhttp://www.blogger.com/profile/14036776134937458948noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-3371313375251795469.post-11129704506979735762011-06-27T09:31:00.001-07:002011-06-27T09:36:50.199-07:00What is Clustered Cache<span class="Apple-style-span" style="border-collapse: separate; color: magenta; font-family: 'Times New Roman'; font-size: small; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: tahoma,verdana,geneva,lucida,'lucida grande',arial,helvetica,sans-serif; font-size: 13px;">ClusteredCache (com.bowman.cardserv.ClusteredCache)<br />
---------------------------------------------------<br />
<br />
Cache-sharing with the csproxy is a powerful tool that can be used in a number of different ways. Before contemplating<br />
the various possible scenarios, the basic cache model needs to be understood.<br />
<br />
Whenever a proxy node receives a new ecm request from a client, it will query the cache to see if this is the first<br />
time this particular request has been received. One of three things can happen:<br />
<br />
1. The ecm has been received before, and a cw reply has already been fetched from a card. This causes an instant hit.<br />
2. The ecm has been received before, but the card transaction is still ongoing and the reply not yet available.<br />
This causes the request to be held in the cache for a maximum of 'max-cache-wait' seco$$$. As soon as a reply is<br />
available all threads waiting for it will be released with their own copy of the cw.<br />
3. The ecm has not been seen before, no card transaction is pending for it. This will place the ecm in a list of pending<br />
requests, and it will be up to the proxy service mapper and connection manager to find a card to handle it.<br />
<br />
The cache knows nothing about which ca-system the ecm belongs to, what sid it is for, or who is asking for it. It will<br />
simply check its lists of already handled ecm -> cw pairs, and the list of pending ecm requests and see which (if any)<br />
contains the new ecm. If two different profiles, or multiple services within one profile, happen to use the same ecm's<br />
the cache will score hits. The chances of a false positive are insignificant enough to be safely ignored.<br />
<br />
When using remote cache-sharing, all of the above remains pretty much the same. However, whenever something is added<br />
to the list of pending requests or requests with cw replies, the cache will broadcast the new addition to the remote<br />
proxy (or proxies) that it has been configured to talk to. This broadcast can be done in several ways, but all involve<br />
udp communication and standard java object serialization.<br />
<br />
Note that in any cache cluster, there can still be occasions when two or more proxies will attempt to query a card for<br />
the same ecm. This will occur when the ecms arrive at both proxies at exactly the same time (which is of course often<br />
the case). Depending on the roundtrip ping time between the proxies, this can be enough to reduce the effectiveness<br />
of the cache significantly. Typically though, you have multiple proxies for the sake of redundancy, you probably want<br />
the same ecm processed in several places so that if one proxy fails to produce a reply in time, another may succeed.<br />
<br />
That said, it is possible to achieve a strict synchronization between cache-instances by using the arbitration feature,<br />
which introduces a negotiation procedure for each ecm, to determine which proxy is best suited to handle it (and then<br />
only that proxy will proceed with forwarding, the others will wait). See sync-period below.<br />
<br />
NOTE: As of 0.8.13 the ClusteredCache no longer uses default java object serialization for the transport protocol.<br />
The new custom protocol is briefly documented in ClusteredCache.java (it should be about 20-40 times more efficient).<br />
<br />
------------------------------------------------------<br />
<br />
The following are settings are available, see proxy.xml for separate examples:<br />
<br />
<cache-handler class="com.bowman.cardserv.ClusteredCache"><br />
<br />
- To use the ClusteredCache as the cache-handler for the proxy, use the class name: com.bowman.cardserv.ClusteredCache<br />
Changing cache-handlers requires a proxy restart.<br />
<br />
<cache-config><br />
<cw-max-age>19</cw-max-age><br />
<max-cache-wait>7</max-cache-wait><br />
<br />
- Inherited from DefaultCache (see proxy-reference.html).<br />
<br />
<remote-host>peer.proxy.host.com</remote-host><br />
<remote-port>54278</remote-port><br />
<br />
- One way of specifying where to send cache-updates, when there is only a single target proxy.<br />
<br />
<multicast-group>230.2.3.2</multicast-group><br />
<multicast-ttl>2</multicast-ttl><br />
<remote-port>54278</remote-port><br />
<br />
- Another way to configure targets for cache-updates. Multicast typically only works in a LAN environment.<br />
<br />
<tracker-url><a href="http://cstracker.host.com/list.enc" style="text-decoration: underline;" target="_blank">http://cstracker.host.com/list.enc</a></tracker-url><br />
<tracker-key>secretkey</tracker-key><br />
<tracker-update>10</tracker-update> <!-- minutes --><br />
<br />
- A third (and perhaps the best) option for configuring multiple targets for cache-updates. The ClusteredCache will<br />
fetch a plain text list of hostnames and portnumbers, and send updates to every entry in the list. The list can be<br />
automatically fetched at regular intervals (or if tracker-update is 0, only when proxy.xml is modified/touched).<br />
This approach allows proxies to be added to a cluster without having to modify the configuration of already existing nodes.<br />
The list must have the following format (# are comments):<br />
<br />
# ClusteredCache list file. Syntax: hostname_or_ip:udp_port_nr<br />
proxy1.host.com:54278<br />
proxy2.host.com:54278<br />
192.168.0.3:54275<br />
<br />
The list can be stored anywhere, as long as it can be accessed via url (e.g file://, http, https, ftp).<span class="Apple-converted-space"> </span><br />
Optionally, the list can also be blowfish encrypted using the included tool fishenc.jar (found in lib, java -jar fishenc.jar).<br />
If encrypted, tracker-key must be correctly set.<br />
<br />
<local-port>54278</local-port><span class="Apple-converted-space"> </span><br />
<br />
- The UDP port where this ClusteredCache instance will listen for incoming updates.<span class="Apple-converted-space"> </span><br />
<br />
<local-host>csproxy3.host.com</local-host><br />
<br />
- The external hostname or IP of this ClusteredCache. This is only needed when using the above tracker setup. The name<br />
should match the one in the tracker list, so the cache instance can identify itself in the list and avoid sending itself<br />
updates.<br />
<br />
<debug>true</debug><br />
<br />
- Set debug to true to enable additional cache information in the status-web. This can impact performance, use with care.<br />
<br />
<hide-names>false</hide-names><br />
<br />
- If configured to send to one or more remote caches, this controls whether the names of the connectors are censored in<br />
the outgoing cache updates (will appear as remote: unknown to the target). Only makes sense when dealing with untrusted<br />
proxy peers.<br />
<br />
<sync-period>0</sync-period> <!-- milliseco$$$ --><br />
<br />
- Set this > 0 to enable the strict arbitration procedure. For example if you set it to 200, the ClusteredCache<br />
would use 200 ms for every new (previously unseen ecm) to wait and synchronize with as many other proxies in the cluster<br />
as possible, and determine who is best suited to handle it. Only this proxy would proceed with a forward to a card.<br />
This adds 200 ms to every single transaction, but should ensure that a cluster of proxies will only ask one card in one<br />
proxy, once, for the same ecm. Probably only usable in a cluster where all nodes are fully trusted, and where the<br />
network is reliable with fairly fixed ping times and no congestion.<br />
<br />
</cache-config><br />
<br />
------------------------------------------------------<br />
<br />
NOTE: It is possible to set up the ClusteredCache without any remote targets (receive-only-mode). If no remote-host/port<br />
is set using any of the various methods then the default behaviour will be to not attempt any sending of updates.<br />
This is useful when creating a cache-only proxy node, receiving updates from multiple other proxies but sending to none.<br />
It can also be used to augment local cards with additional services (which will only be available through the cache).<br />
<br />
With 0.9.0, ClusteredCache in receive-only mode is the default in the auto-generated config template. If nothing is<br />
received, it behaves exactly like the DefaultCache.</span></span>CSPhttp://www.blogger.com/profile/14036776134937458948noreply@blogger.comtag:blogger.com,1999:blog-3371313375251795469.post-58420807863857515432011-06-26T13:45:00.000-07:002011-06-26T13:46:14.453-07:00CSP Optimization<div style="color: magenta;"><b><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Times New Roman'; font-size: small; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: verdana,geneva,lucida,'lucida grande',arial,helvetica,sans-serif; font-size: 13px;"><br />
Contents:<br />
1. Client behaviour<br />
2. Cardserver behaviour<br />
3. Proxy configuration<br />
4. Interpreting transaction flags<br />
5. Linux/OS-tweaking<br />
6. JVM tweaking<br />
<br />
There are a considerable number of timeout values at play for any given ecm -> cw transaction, this readme will attempt<br />
to identify the most common timing pitfalls and suggest a set of best practices.<br />
<br />
It should be said that by far the greatest increase to stability comes from adding a second proxy, compared to that<br />
any other optimizations are insignificant. Even if both proxies are connected to the exact same set of cards, the<br />
second chance it offers to clients (regardless of what the issue was) has a _major_ impact on the perceived stability.<br />
This is of course assuming clients with failover functionality similar to mgcamd, that maintain all configured connections<br />
(with keep-alives) and will perform a seamless switch to the next applicable connection in case of trouble.<br />
<br />
NOTE: Most of the tips here are based on ca-systems where the possible delay between the ecm appearing in the TS and<br />
the cw actually changing is fairly large (>5 seconds). For ca-systems with tighter margins you may need other methods.<br />
In most proxy examples, it is assumed that the client can safely wait nearly 9 seconds for its cw reply without<br />
experiencing a freeze.<br />
<br />
<br />
1. Client behaviour (most of the proxy examples are based on mgcamd, or similar clients).<br />
-----------------------------------------------------------------------------------------<br />
<br />
- Identify and configure the ecm timeout* value for each client, and set this as high as possible.<span class="Apple-converted-space"> </span><br />
I.e. if the situation allows waiting for 9 seconds without causing a picture freeze, then set the value to 9 seconds.<span class="Apple-converted-space"> </span><br />
Experiment to determine what the actual maximum is (you can use the LoggingPlugin, see 3. Proxy configuration).<br />
If you have multiple proxies or other alternative connections, set the timeout slightly below the maximum to give the<br />
client a chance to get a reply elsewhere in case of failure.<br />
<br />
- Client ecm cache and reconnect/retry behaviour etc, are less important and can usually be safely ignored.<span class="Apple-converted-space"> </span><br />
If possible the client should be configured so that it tries to connect to a previously unconnected server each time<span class="Apple-converted-space"> </span><br />
it sees a new ecm.<br />
<br />
- Always keep full client logging enabled when trying to determine the effects of configuration changes.<br />
Usually its a good idea to always have udp logging enabled (with any and all debug options), and send the packages<br />
to whatever desktop machine you're at. Then when troubleshooting is required, all you need to do is start a listener,<br />
for example netcat: nc -l -u -p portnumber<br />
On windows, netcat is available via cygwin (cygwin.com).<br />
<br />
* Ecm timeout here refers to the maximum time that the client will wait for a cw response, after sending the ecm request<br />
to the server, before giving up and moving on (typically resulting in a freeze). For example: the "K" value in mg_cfg.<br />
<br />
<br />
2. Cardserver behaviour (most of the proxy examples assume newcs, but mpcs/oscam or rqcs have also been used).<br />
--------------------------------------------------------------------------------------------------------------<br />
<br />
- Ensure that the server does not have any unnecessary limitations imposed on the account used by the proxy.<br />
Rate limits are of course especially harmful and should be removed completely.<span class="Apple-converted-space"> </span><br />
<br />
- Avoid configurations where the server attempts to connect back to the client (a newcamd feature). I.e. if the server<br />
allows specifying hostname/port per user account, then make sure you don't. Remove any such elements.<br />
<br />
- Ecm cache and emm cache can use reasonable defaults, there is no benefit in modifying them for the sake of the proxy.<br />
Typically the ecm cache would not be used at all, and the emm cache is only there to prevent unecessary time consuming<br />
card communication.<br />
<br />
- Priority in the ecm queue is a non-issue when the server only has proxy users, but should be left at default or set<br />
to fifo (or whatever setting causes requests to be processed in the order of arrival).<br />
<br />
- Again, keep full logging enabled while troubleshooting. If the server has udp logging, use it to send the log<span class="Apple-converted-space"> </span><br />
events to the same target machine used by the client logging (use a different port though) so you can compare both<span class="Apple-converted-space"> </span><br />
side by side in real time.<br />
<br />
- Keep an eye on cpu use on the server, especially if it is running on a small embedded system. If it is maxed out<br />
it is more likely to experience glitches (returning duplicates, timeouts, card communication failures etc). If so it<br />
may be necessary to insert an artifical delay between asynch requests (see default-min-delay in proxy-reference.html).<br />
<br />
<br />
3. Proxy configuration<br />
----------------------<br />
<br />
Always start from scratch with the auto-generated config. Do not add any options until you understand what they do and<br />
you see an actual need from analyzing the logs.<br />
<br />
To get a definitive overview of what the proxy does with each config value (and where it does it), you can start java<br />
with the following param: -Dcom.bowman.cardserv.util.tracexmlcfg=true<br />
That will track all access to proxy.xml and when you request it (via the admin section of the status web), dump the<br />
trace to file in the proxy etc dir.<br />
<br />
It will produce output such as this:<br />
<br />
/cardserv-proxy/connection-manager<br />
cannot-decode-wait<br />
type: Time (defSuffix=s)<br />
default: '0'<br />
caller: CwsConnectorManager.configUpdated(CwsConnectorMana ger.java:63)<br />
<br />
Which means that the CwsConnectorManager checks the value of the setting <cannot-decode-wait> on line 63 in the method<br />
configUpdated() in the file CwsConnectorManager.java. It also shows (as of 0.9.0) that this is a time value and that<br />
the default is 0 (disabled in this case), as well as specified in seconds unless otherwise indicated.<br />
<br />
<br />
- Asynchronous vs synchronous newcamd<br />
<br />
With synchronous newcamd communication (default) the proxy will only send one request at a time to each connector, and<br />
await a reply to that before proceeding. In theory this is inefficient, since the newcamd protocol supports multiple<br />
concurrent requests. In synchronous mode any and all delays that occur due to network latency spikes (or other glitches)<br />
or simply other users on the server side, will delay the entire proxy queue for the connector.<br />
<br />
In the asynchronous case, the proxy sends all pending requests for the connector immediately to the server as fast as it<br />
will read them. The replies are then matched to requests by the newcamd sequence id present in each message, as they<br />
come back.<br />
This effectively moves the queue handling from the proxy to the server, and should mean a higher throughput as there is<br />
less room for network interference. Not all servers support async, and even those that do (e.g later newcs versions)<span class="Apple-converted-space"> </span><br />
seem to sometimes return unexpected sequence ids. Situations where multiple proxies (or other users) use the same newcs<br />
port seem especially error prone.<br />
<br />
As a rule of thumb, if you want to get the maximum capacity from your connectors you should try async mode. Watch the<br />
logs closely for issues related to out of sequence data though (unexpected/unknown replies and subsequent timeouts),<br />
or it could end up doing more harm then good.<br />
<br />
If a newcamd-connector points to another csproxy, it should use synchronous mode (as the proxy will assume each client<span class="Apple-converted-space"> </span><br />
is a regular user which will have at most one pending ecm request at any given time). The csp-connector type (0.9.0+) is<span class="Apple-converted-space"> </span><br />
async by default.<br />
<br />
<br />
- The following timing values are significant:<br />
<br />
<connection-manager> / <max-cw-wait><br />
Set this to the same as the client ecm timeout (e.g 9 seconds) or if in doubt, leave it at the default setting.<br />
Changing it affects what the proxy considers to be one cw validity period, and this in turn affects statistics and<span class="Apple-converted-space"> </span><br />
capacity estimates.<span class="Apple-converted-space"> </span><br />
Under normal circumstances, you never want to set this value below the ecm timeout of the clients (it should be<br />
equal or larger). In general however, it depends on which timeout you wish to be reached first, client or proxy.<br />
It may depend on which clients are in use and how they deal with timeouts.<br />
<br />
To help figure out the exact cutoff point for clients (the maximum time they can wait before they experience a freeze),<br />
use the LoggingPlugin and the set-test-delay ctrl command (via the status web). This allows you to experiment by adding<br />
a gradually increasing delay for a specific test-client (use the ip filter). Watch the client logs closely and try to<br />
see at which exact ecm round-trip time the freezes start occuring. This is your max-delay for this particular ca-system.<br />
If the time is very short (<3 seconds) it might not be possible to set the max-cw-wait for the proxy to the same, but<br />
at least you know where the cutoff point is. Values <4 have not been tested with the proxy, but may still work.<br />
NOTE: To get accurate values with test-delay, use a service that is cached (that other clients are currently watching).<br />
Also, make sure the client ecm timeout is increased to a level where it will not interfere before the test (>9 secs).<br />
That means you cannot use clients such as CCcam where the newcamd timeout is locked to 4 seconds and cannot be changed.<br />
<br />
<connection-manager> / <congestion-limit><br />
This allows you to configure the point at which the proxy considers a connector to be congested. The value refers<br />
to the estimated connector queue-length (in seconds). By default it is the same as the <max-cw-wait> but its usually<br />
a good idea to set it slightly below, to force service-mapper and metric handling to move to other connectors sooner.<br />
I.e: if max-cw-wait is 5, set congestion limit to 4 (causing any connector with a queue of >8 to be flagged as congested).<br />
Using this mainly makes sense if you have multiple cards with different metric levels and want to fine-tune when the<br />
higher metric cards start getting used.<br />
<br />
<cache-handler> / <cache-config> / <max-cache-wait><br />
If card capacity is a problem, you want to keep this at a level that is as high as possible but still allows room for<br />
one retry before the max-cw-wait is reached.<br />
I.e. if max-cw-wait is 9, and the average response time for your profiles is around 2 secs, set the max-cache-wait to 6<br />
or 7 secs.<br />
This should result in a behaviour where a timeout in the cache means the client can still receive a reply before<br />
reaching its own timeout. Use the transaction flags to determine whether this is actually the case (see below).<br />
<br />
Cache timeouts occur when an expected reply never arrives, either because the server that the request was sent to<br />
became disconnected or overloaded, or because a remote cache that indicated it was going to provide the reply failed to<br />
do so (or did but it got lost in transit). It is normal to see more cache timeouts in a proxy setup that uses the<br />
clustered cache.<br />
<br />
For most situations, you want a max-cache-wait that is just slightly above the worst case processing time (if its<br />
2500 ms, you might set 3000 ms as the cache wait). This ensures that you only get cache timeouts when actual problems<br />
occur, not because normal processing is taking longer than usual (due to peak load or network latency).<br />
As of 0.9.0, it is possible to configure max-cache-wait with a percentage string, e.g "50%". This allows for different<br />
wait times depending on the max-cw-wait in effect for the profile. If you have multiple profiles with significantly<br />
different max-cw-wait times then you should definately use a percentage setting.<br />
<br />
<br />
- These are less important but deserve mention:<br />
<br />
<connection-manager> / <reconnect-interval><br />
This only determines how often the proxy will attempt to reconnect to disconnected servers. It can be reduced to<br />
10-15 seconds (but avoid lower values) to minimize downtime due to shaky connections when there are few cards available.<br />
Keep in mind that whenever a connection is lost one immediate retry is performed, so its usually best to leave this<br />
at 30-60 seconds. Setting too low may cause connectors to never reconnect.<br />
<br />
<connection-manager> / <default-keepalive-interval><br />
If there are nat-routers or some equivalent between proxy and server, tcp's that are idle for more than a set time<br />
may end up getting disconnected. Setting this value to slightly below that idle timeout will ensure they stay up.<span class="Apple-converted-space"> </span><br />
E.g: 59 seconds. Note that no keep-alives are sent as long as there is traffic on a server connection, so<br />
there is no danger of flooding it with unnecessary messages.<br />
<br />
<br />
- Don't touch these unless you're experimenting with some really radical cache-sharing setups:<br />
<br />
<connection-manager> / <cannot-decode-wait><br />
<cache-handler> / <cache-config> / <cw-max-age><br />
<br />
<br />
- For very large shares changing these may be required (see proxy-reference.html):<br />
<br />
<ca-profiles> / <max-threads><br />
<ca-profiles> / <sesion-timeout><br />
<connection-manager> / <default-max-queue> (also available as <max-queue> set per connector)<br />
<connection-manager> / <default-min-delay> (also available as <min-delay> set per connector)<br />
<br />
<br />
- For proxy troubleshooting, make sure you have debug="true" for all profiles, and logging configured as follows:<br />
<br />
<logging log-ecm="true" log-emm="false" log-zapping="false" hide-ip-addresses="false"><br />
<log-file>log/cardserv.log</log-file><br />
<log-level>FINE</log-level><br />
<silent>true</silent><br />
<debug>true</debug><br />
<warning-threshold bad-flags="YNTSOGXWD-" max-delay="6000"/><br />
<event-threshold min-count="1"/><br />
</logging><br />
<br />
In general this is a reasonable troubleshooting default, but of course depending on what you're looking for it may need<br />
changing. Under normal operation you want to use level INFO (or WARNING), and it may make sense to only log transactions<br />
exceeding the client ecm timeout limit as warnings. I.e. use something like:<br />
<warning-threshold bad-flags="YTSAQGXWD-" max-delay="8000"/><br />
<br />
For each transaction that meets the warning criteria, a more detailed time log will be made available, showing exactly<br />
what the wait time was spent on.<span class="Apple-converted-space"> </span><br />
- Cache wait: The time the request was locked in the cache, waiting for another transaction forwarding to a card.<br />
- Send queue: Time spent waiting in line to be sent to a card (usually 0 in async mode as everything is sent immediately).<br />
- Server wait: Time spent waiting for the server to reply. For normal (cache miss) transactions this should be longest.<br />
- Client writeback: Time spent sending the reply back to the client. Typically 0-1 ms, unless there is a network issue.<br />
<br />
<br />
4. Interpreting transaction flags<br />
---------------------------------<br />
<br />
Normal traffic flags:<span class="Apple-converted-space"> </span><br />
F = Normal forward to CWS<br />
C = Cache hit (local)<br />
R = Cache hit (received from remote cache)<br />
I = Instant cache hit (no waiting at all in cache, both request and reply already available)<br />
1 = This was the first transaction performed by a new session<br />
Z = SID changed (compared to previous transaction = user zap, can also indicate multiple tuners or users in one session if it occurs every time)<br />
<br />
Service mapper flags:<br />
N = Cannot decode (service mapper says service not on any card, or service blocked)<br />
P = Service mapper didn't know status for this SID on one or more cards (may have triggered probing if accepted by connectors)<br />
2 = Triggered broadcast to additional cards besides the selected one (broadcast-missing-sid or redundant-forwarding in use)<br />
+ = Caused an addition to the service map (found service, after probing/successful decode)<br />
- = Caused a removal from the service map (lost service, after repeated failed decodes)<br />
<br />
Flags indicating possible problems/recovery situations:<br />
B = Blocked by exceeded limits or by filters (plugins)<br />
M = Ca-id mismatch. The ecm reply didn't have the same ca-id as the request, indicates some clients are sending ecms to the wrong ports.<br />
E = Client got an empty reply (cannot-decode received from CWS, or from the proxy itself with situation N<br />
Y = Forward retry performed (first chosen CWS disconnected during the transaction, but alternatives exist)<br />
A = Abort when forwarding (CWS connection was closed before forward, because of other request timeouts or by the network/server)<br />
T = Timeout when forwarding (no response from CWS within time limit, i.e max-cw-wait)<br />
O = Timeout while waiting in cache (waited for max-cache-wait, but no reply was reported to the cache)<br />
Q = Abort while waiting in cache (the forward the cache was waiting for failed, either locally or remotely)<br />
G = Congestion when forwarding (time was > max-cw-wait/2, but forward still performed)<br />
X = Cache hit after failed forward (situation <strong>Y</strong>, but reply became available in cache so no need for new forward)<br />
<br />
Internal/debugging flags:<br />
S = Timeout in send queue (when trying to forward to connector, should normally not occur)<br />
W = Triggered cannot-decode-wait (would have been situation <strong>N</strong>, but waiting for remote cache paid off)<br />
H = Caused an internal failed rechecking of the cache, represents an attempt to recover from an immediately preceeding problem<br />
U = Interrupt in the no-sid-delay sleep, a request without sid was delayed but disconnected during the delay<br />
D = The user session disconnected before it could receive the reply (likely reached the client ecm timeout)<br />
<br />
<br />
Transaction flags are listed in the order in which they were set internally by csp (before 0.8.7 it was alphabetical).<br />
I.e. "OFD" reads as: cache timeout occured (O), so attempted forward to card (which succeeded - F), but when the<br />
reply from the forward was returned the client had already disconnected (D).<br />
Note that many flags can occur at virtually the same time, so the order isn't necessarily chronological in all cases.<br />
<br />
Examples of common transactions (with the flags ordered as of before 0.8.7):<br />
<br />
"FPXZ" - User zapped (Z) to a service that the service mapper lacked knowledge for on one or more cards, so<span class="Apple-converted-space"> </span><br />
probes were sent to multiple cards (P). Not knowing where to find the service, the proxy forwarded to the least loaded<br />
card in the profile (F). This card failed to provide a reply, but after the failure a reply was found in the cache (X).<br />
I.e. one of the other probes did get a successful reply.<br />
<br />
"+FPZ" - Same as above, except the least loaded first choice was able to provide a reply, and the service mapper used<br />
this information to update the mapped status for this card (+).<br />
<br />
"FO" - A cache timeout occured (O, remote cache update may have failed to arrive in time. There was time left before<br />
max-cw-wait, so a forward was performed (and suceeded, F).<br />
<br />
"CIZ" - User zapped (Z) to a service already present in cache (C), and immediately received a reply with no waiting (I).<br />
"IRZ" - Same except the cache entry came from a remote cache.<br />
<br />
"FOZ" - The first transaction for a new client connection (Z) resulted in a cache timeout (O) followed by a forward (F).<br />
Normal if it occurs mainly in connection with proxy restarts (local or remote), before all cards have been connected.<br />
<br />
"ENZ" - User zapped (Z) to a blocked service causing the proxy to immediately reply with an empty cannot-decode (EN).<br />
<br />
"+NPZ" - Same as above, except the service wasn't blocked, only marked as non-existant on all cards by the service<br />
mapper. This one request meant the auto-reset-threshold was exceeded, so probes were sent (P) and the first one got<br />
a successful reply, causing the mapper to change the status on that card from cannot decode to can decode (+).<br />
<br />
"-EFZ" - User zapped (Z) to a channel believed to exist on a card, but when it was forwarded the server replied with a<br />
cannot decode (EF) and this was the 2nd such failure for this card, so the service mapper removed the channel (-).<br />
<br />
"EFT" - A forward (F) timed out waiting for the reply to be returned from the server (T), and there was no time left<br />
for retries so an empty (cannot decode, E) reply was sent back to the client.<span class="Apple-converted-space"> </span><br />
<br />
This scenario can be fairly common and would show something like this in the standard log, for a situation where the<span class="Apple-converted-space"> </span><br />
reply was just a little too late:<br />
<br />
WARNING -> NewcamdCws[cwsname<img alt="" border="0" class="inlineimg" src="http://www.cccamforum.com/forums/images/smilies/tongue.gif" style="vertical-align: middle;" title="Stick Out Tongue" />rofile] <- Timeout waiting for ecm reply, discarding request and returning empty<span class="Apple-converted-space"> </span><br />
(1 failures) - ECM B9328A35 - [16] - Waited: 8241ms<br />
^^ sequence id<br />
INFO -> NewcamdCws[cwsname<img alt="" border="0" class="inlineimg" src="http://www.cccamforum.com/forums/images/smilies/tongue.gif" style="vertical-align: middle;" title="Stick Out Tongue" />rofile] <- Message received, cancelling timeout-state<br />
<br />
WARNING -> NewcamdCws[cwsname<img alt="" border="0" class="inlineimg" src="http://www.cccamforum.com/forums/images/smilies/tongue.gif" style="vertical-align: middle;" title="Stick Out Tongue" />rofile] <- No request found for reply: 16 (DCW, ServiceName profile:sid)<span class="Apple-converted-space"> </span><br />
^^ sequence id<br />
I.e. when the reply is finally received, the request is already deleted and the client has been given a failure reply.<br />
<br />
<br />
5. Linux/OS-tweaking<br />
--------------------<br />
<br />
As of 0.8.11 the proxy will use sun java6 features to keep track of used and available unix file descriptors (file<br />
handles). This is a common source of problems, as each tcp socket uses at least one and the default maximum limit per<br />
user is commonly 1024. If exceeded there would be java exception stacktraces in the logs with messages such as<br />
"too many open files".<br />
<br />
You can check this by logging in as the user who will be running the proxy and doing:<br />
$ ulimit -n<br />
1024<br />
<br />
To increase the limit you typically have to edit /etc/security/limits.conf and add lines like these:<br />
proxyuser soft nofile 8192<br />
proxyuser hard nofile 8192<br />
<br />
After a reboot ulimit -n output should show 8192 when logging in as proxyuser. If running sun java6, the currently used<span class="Apple-converted-space"> </span><br />
and available filedescriptors will be visible in the status web (as part of the proxy-status xml api command) shown as:<br />
FD: [used/maximum]<br />
<br />
If not using sun java6, you can still get some idea how many handles are used by the proxy using lsof for the proxy pid.<br />
For example:<br />
$ /usr/sbin/lsof -p `cat cardservproxy.pid` | wc -l<br />
181<br />
<br />
Note that this value will tend to be higher by a fixed number than the one reported by the sun jvm, presumably because<br />
lsof also lists some file handles that don't count against the per-user quota.<span class="Apple-converted-space"> </span><br />
I.e. when lsof shows 181, FD would show 141 (the actual number that will cause problems when it reaches the limit).<br />
<br />
Depending on OS and JVM, there might also be other limits hit (such as noproc, number of processes/threads).<br />
<br />
<br />
6. JVM tweaking<br />
---------------<br />
<br />
As mentioned elsewhere the proxy only works properly with real sun java (and was only tested under 1.4.2 and 1.6).<br />
To verify that you're using the correct one, check the status web. It should mention HotSpot if it is sun, i.e:<br />
[Java HotSpot(TM) Server VM1.6.0_03]<br />
Server or client doesn't matter much, and neither does 64/32 bit (although 64 will tend to use significantly more heap).<br />
If it turns out to be any other jvm implementation you're likely to run into subtle and sporadic problems.<br />
If you've found another that does seem to run the proxy well enough under load, and you can prove it: let me know.<br />
As of 0.9.0, a super user can click on the jvm name in the status web to trigger the status command 'system-properties',<br />
which shows all the jvm information as xml.<br />
<br />
The Heap usage displayed on the status web shows the amount of memory currently in use (within this jvm instance, it<br />
says nothing about the rest of the system) and the total allocated by the jvm so far, i.e:<br />
[12503k/57536k] means ~12 megs in use and ~57 peak allocation. It does not imply that 57 is the maximum available to<br />
the jvm, once it is reached it will probably be able to allocate more.<br />
If it is unable to do so you'll start to get OutOfMemoryError in your logs (hopefully, it may just fail silently).<br />
The maximum heap the jvm is allowed to allocate by default depend on the jvm version and os.<br />
To increase it, add the following parameter to the line that starts java (in cardproxy.sh if you use that script):<br />
-Xmx256m<br />
That would allow the heap to grow up to 256 megs, which should be more than enough for most use cases.<br />
If you know how much your setup usually ends up using, you can give the jvm that much to start with, by also adding:<br />
-Xms100m (for example, this will allocate 100 megs immediately on startup).<br />
<br />
The next value 'TC' is for thread count. This will normally be a fixed number around 15, + 2 for every connector, + 1<br />
for every listen port, + 1 for every client session. Use it to spot problems. As long as none of the above changes,<br />
the thread count shouldn't either. If it does then something is going on that you should look into.<br />
As of 0.9.0, clicking on the count will show a list of the actual threads. All threads in the proxy core use very<br />
descriptive names so it should be immediately apparent what each one does.</span></span></b></div>CSPhttp://www.blogger.com/profile/14036776134937458948noreply@blogger.comtag:blogger.com,1999:blog-3371313375251795469.post-44447047446663885512011-06-24T12:20:00.000-07:002011-06-24T12:20:31.518-07:00How to : Install Sun Java (JDK) On linux Pc To Run CSP<div style="color: magenta;"><b><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: Verdana,'Trebuchet MS','Lucida Grande',Helvetica,Arial,sans-serif; font-size: 13px; line-height: 18px;">to Run card server proxy ....<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />Requirements : Desktop Pc Running On Linux , Ram At Least 512 MB , Fair Free Disk Space .<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />this tutorial tested on : Cent O/s , Fedora , Ubuntu<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />installation of operating system will be explained in other topics .<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />now we go to<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><a class="postlink" href="http://java.sun.com/javase/downloads/index.jsp" style="margin: 0px; padding: 0px; text-decoration: none;">http://java.sun.com/javase/downloads/index.jsp</a></span></span></b></div><div style="color: magenta;"><b><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: Verdana,'Trebuchet MS','Lucida Grande',Helvetica,Arial,sans-serif; font-size: 13px; line-height: 18px;">nd choose :</span></span></b></div><div style="color: magenta;"><b><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: Verdana,'Trebuchet MS','Lucida Grande',Helvetica,Arial,sans-serif; font-size: 13px; line-height: 18px;">You Will Have A Screen , Choose Your operating system and click continue .</span></span></b></div><div style="color: magenta;"><b><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: Verdana,'Trebuchet MS','Lucida Grande',Helvetica,Arial,sans-serif; font-size: 13px; line-height: 18px;">A Window Will Appear , Chose Skip This Step.</span></span></b></div><div style="color: magenta;"><b><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: Verdana,'Trebuchet MS','Lucida Grande',Helvetica,Arial,sans-serif; font-size: 13px; line-height: 18px;">now download the file from the link to your PC ,<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />if you are connected to A Remote Server .<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />write the Order :<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />Quote:<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />#root@local : wget ( the link appeared in the window )<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />after downloading is done.<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />it`s time to install Java.<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />write those Orders :<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />Quote:<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />cd /<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />chmod a+x ( downloade file name )<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />after that<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />Quote:<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />./filename.rpm.bin</span></span></b></div><div style="color: magenta;"><b><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: Verdana,'Trebuchet MS','Lucida Grande',Helvetica,Arial,sans-serif; font-size: 13px; line-height: 18px;">Click on space to scroll down to the end of the license agreement, then type in yes and let the installer unzip the RPM files and perform the installation.</span></span></b></div><b style="color: magenta;"><span class="Apple-style-span" style="border-collapse: separate; font-family: 'Times New Roman'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span class="Apple-style-span" style="font-family: Verdana,'Trebuchet MS','Lucida Grande',Helvetica,Arial,sans-serif; font-size: 13px; line-height: 18px;">After the installation has been done you can delete installer and the temporary files it created on your desktop by issuing the following commands:<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" /><br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />Quote:<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />rm jdk-6u2-linux*<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />rm sun-java*.rpm<br style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;" />Type y when prompted to delete each file (make sure you are not deleting something else).</span></span></b>CSPhttp://www.blogger.com/profile/14036776134937458948noreply@blogger.com