Class MCollective::RPC::Client
In: lib/mcollective/rpc/client.rb
Parent: Object

The main component of the Simple RPC client system, this wraps around MCollective::Client and just brings in a lot of convention and standard approached.

Methods

Attributes

agent  [R] 
batch_mode  [R] 
batch_size  [R] 
batch_sleep_time  [R] 
client  [R] 
config  [RW] 
ddl  [R] 
discovery_method  [R] 
discovery_options  [R] 
filter  [RW] 
limit_method  [R] 
limit_targets  [R] 
output_format  [R] 
progress  [RW] 
reply_to  [RW] 
stats  [R] 
timeout  [RW] 
ttl  [RW] 
verbose  [RW] 

Public Class methods

Creates a stub for a remote agent, you can pass in an options array in the flags which will then be used else it will just create a default options array with filtering enabled based on the standard command line use.

  rpc = RPC::Client.new("rpctest", :configfile => "client.cfg", :options => options)

You typically would not call this directly you‘d use MCollective::RPC#rpcclient instead which is a wrapper around this that can be used as a Mixin

[Source]

     # File lib/mcollective/rpc/client.rb, line 20
 20:       def initialize(agent, flags = {})
 21:         if flags.include?(:options)
 22:           initial_options = flags[:options]
 23: 
 24:         elsif @@initial_options
 25:           initial_options = Marshal.load(@@initial_options)
 26: 
 27:         else
 28:           oparser = MCollective::Optionparser.new({:verbose => false, :progress_bar => true, :mcollective_limit_targets => false, :batch_size => nil, :batch_sleep_time => 1}, "filter")
 29: 
 30:           initial_options = oparser.parse do |parser, opts|
 31:             if block_given?
 32:               yield(parser, opts)
 33:             end
 34: 
 35:             Helpers.add_simplerpc_options(parser, opts)
 36:           end
 37: 
 38:           @@initial_options = Marshal.dump(initial_options)
 39:         end
 40: 
 41:         @initial_options = initial_options
 42:         @stats = Stats.new
 43:         @agent = agent
 44:         @timeout = initial_options[:timeout] || 5
 45:         @verbose = initial_options[:verbose]
 46:         @filter = initial_options[:filter]
 47:         @config = initial_options[:config]
 48:         @discovered_agents = nil
 49:         @progress = initial_options[:progress_bar]
 50:         @limit_targets = initial_options[:mcollective_limit_targets]
 51:         @limit_method = Config.instance.rpclimitmethod
 52:         @output_format = initial_options[:output_format] || :console
 53:         @force_direct_request = false
 54:         @reply_to = initial_options[:reply_to]
 55:         @discovery_method = initial_options[:discovery_method]
 56:         @discovery_options = initial_options[:discovery_options] || []
 57: 
 58:         @batch_size = Integer(initial_options[:batch_size] || 0)
 59:         @batch_sleep_time = Float(initial_options[:batch_sleep_time] || 1)
 60:         @batch_mode = @batch_size > 0
 61: 
 62:         agent_filter agent
 63: 
 64:         @client = MCollective::Client.new(@config)
 65:         @client.options = initial_options
 66: 
 67:         @discovery_timeout = discovery_timeout
 68: 
 69:         @collective = @client.collective
 70:         @ttl = initial_options[:ttl] || Config.instance.ttl
 71: 
 72:         # if we can find a DDL for the service override
 73:         # the timeout of the client so we always magically
 74:         # wait appropriate amounts of time.
 75:         #
 76:         # We add the discovery timeout to the ddl supplied
 77:         # timeout as the discovery timeout tends to be tuned
 78:         # for local network conditions and fact source speed
 79:         # which would other wise not be accounted for and
 80:         # some results might get missed.
 81:         #
 82:         # We do this only if the timeout is the default 5
 83:         # seconds, so that users cli overrides will still
 84:         # get applied
 85:         begin
 86:           @ddl = DDL.new(agent)
 87:           @timeout = @ddl.meta[:timeout] + @discovery_timeout if @timeout == 5
 88:         rescue Exception => e
 89:           Log.debug("Could not find DDL: #{e}")
 90:           @ddl = nil
 91:         end
 92: 
 93:         # allows stderr and stdout to be overridden for testing
 94:         # but also for web apps that might not want a bunch of stuff
 95:         # generated to actual file handles
 96:         if initial_options[:stderr]
 97:           @stderr = initial_options[:stderr]
 98:         else
 99:           @stderr = STDERR
100:           @stderr.sync = true
101:         end
102: 
103:         if initial_options[:stdout]
104:           @stdout = initial_options[:stdout]
105:         else
106:           @stdout = STDOUT
107:           @stdout.sync = true
108:         end
109:       end

Public Instance methods

Sets the agent filter

[Source]

     # File lib/mcollective/rpc/client.rb, line 376
376:       def agent_filter(agent)
377:         @filter["agent"] << agent
378:         @filter["agent"].compact!
379:         reset
380:       end

Sets the batch size, if the size is set to 0 that will disable batch mode

[Source]

     # File lib/mcollective/rpc/client.rb, line 568
568:       def batch_size=(limit)
569:         raise "Can only set batch size if direct addressing is supported" unless Config.instance.direct_addressing
570: 
571:         @batch_size = Integer(limit)
572:         @batch_mode = @batch_size > 0
573:       end

[Source]

     # File lib/mcollective/rpc/client.rb, line 575
575:       def batch_sleep_time=(time)
576:         raise "Can only set batch sleep time if direct addressing is supported" unless Config.instance.direct_addressing
577: 
578:         @batch_sleep_time = Float(time)
579:       end

Sets the class filter

[Source]

     # File lib/mcollective/rpc/client.rb, line 352
352:       def class_filter(klass)
353:         @filter["cf_class"] << klass
354:         @filter["cf_class"].compact!
355:         reset
356:       end

Sets the collective we are communicating with

[Source]

     # File lib/mcollective/rpc/client.rb, line 533
533:       def collective=(c)
534:         raise "Unknown collective #{c}" unless Config.instance.collectives.include?(c)
535: 
536:         @collective = c
537:         @client.options = options
538:         reset
539:       end

Set a compound filter

[Source]

     # File lib/mcollective/rpc/client.rb, line 390
390:       def compound_filter(filter)
391:         @filter["compound"] <<  Matcher.create_compound_callstack(filter)
392:         reset
393:       end

Constructs custom requests with custom filters and discovery data the idea is that this would be used in web applications where you might be using a cached copy of data provided by a registration agent to figure out on your own what nodes will be responding and what your filter would be.

This will help you essentially short circuit the traditional cycle of:

mc discover / call / wait for discovered nodes

by doing discovery however you like, contructing a filter and a list of nodes you expect responses from.

Other than that it will work exactly like a normal call, blocks will behave the same way, stats will be handled the same way etcetc

If you just wanted to contact one machine for example with a client that already has other filter options setup you can do:

puppet.custom_request("runonce", {}, ["your.box.com"], {:identity => "your.box.com"})

This will do runonce action on just ‘your.box.com’, no discovery will be done and after receiving just one response it will stop waiting for responses

If direct_addressing is enabled in the config file you can provide an empty hash as a filter, this will force that request to be a directly addressed request which technically does not need filters. If you try to use this mode with direct addressing disabled an exception will be raise

[Source]

     # File lib/mcollective/rpc/client.rb, line 279
279:       def custom_request(action, args, expected_agents, filter = {}, &block)
280:         @ddl.validate_rpc_request(action, args) if @ddl
281: 
282:         if filter == {} && !Config.instance.direct_addressing
283:           raise "Attempted to do a filterless custom_request without direct_addressing enabled, preventing unexpected call to all nodes"
284:         end
285: 
286:         @stats.reset
287: 
288:         custom_filter = Util.empty_filter
289:         custom_options = options.clone
290: 
291:         # merge the supplied filter with the standard empty one
292:         # we could just use the merge method but I want to be sure
293:         # we dont merge in stuff that isnt actually valid
294:         ["identity", "fact", "agent", "cf_class", "compound"].each do |ftype|
295:           if filter.include?(ftype)
296:             custom_filter[ftype] = [filter[ftype], custom_filter[ftype]].flatten
297:           end
298:         end
299: 
300:         # ensure that all filters at least restrict the call to the agent we're a proxy for
301:         custom_filter["agent"] << @agent unless custom_filter["agent"].include?(@agent)
302:         custom_options[:filter] = custom_filter
303: 
304:         # Fake out the stats discovery would have put there
305:         @stats.discovered_agents([expected_agents].flatten)
306: 
307:         # Handle fire and forget requests
308:         #
309:         # If a specific reply-to was set then from the client perspective this should
310:         # be a fire and forget request too since no response will ever reach us - it
311:         # will go to the reply-to destination
312:         if args[:process_results] == false || @reply_to
313:           return fire_and_forget_request(action, args, custom_filter)
314:         end
315: 
316:         # Now do a call pretty much exactly like in method_missing except with our own
317:         # options and discovery magic
318:         if block_given?
319:           call_agent(action, args, custom_options, [expected_agents].flatten) do |r|
320:             block.call(r)
321:           end
322:         else
323:           call_agent(action, args, custom_options, [expected_agents].flatten)
324:         end
325:       end

Disconnects cleanly from the middleware

[Source]

     # File lib/mcollective/rpc/client.rb, line 112
112:       def disconnect
113:         @client.disconnect
114:       end

Does discovery based on the filters set, if a discovery was previously done return that else do a new discovery.

Alternatively if identity filters are given and none of them are regular expressions then just use the provided data as discovered data, avoiding discovery

Discovery can be forced if direct_addressing is enabled by passing in an array of nodes with :nodes or JSON data like those produced by mcollective RPC JSON output using :json

Will show a message indicating its doing discovery if running verbose or if the :verbose flag is passed in.

Use reset to force a new discovery

[Source]

     # File lib/mcollective/rpc/client.rb, line 422
422:       def discover(flags={})
423:         flags.keys.each do |key|
424:           raise "Unknown option #{key} passed to discover" unless [:verbose, :hosts, :nodes, :json].include?(key)
425:         end
426: 
427:         flags.include?(:verbose) ? verbose = flags[:verbose] : verbose = @verbose
428: 
429:         verbose = false unless @output_format == :console
430: 
431:         # flags[:nodes] and flags[:hosts] are the same thing, we should never have
432:         # allowed :hosts as that was inconsistent with the established terminology
433:         flags[:nodes] = flags.delete(:hosts) if flags.include?(:hosts)
434: 
435:         reset if flags[:nodes] || flags[:json]
436: 
437:         unless @discovered_agents
438:           # if either hosts or JSON is supplied try to figure out discovery data from there
439:           # if direct_addressing is not enabled this is a critical error as the user might
440:           # not have supplied filters so raise an exception
441:           if flags[:nodes] || flags[:json]
442:             raise "Can only supply discovery data if direct_addressing is enabled" unless Config.instance.direct_addressing
443: 
444:             hosts = []
445: 
446:             if flags[:nodes]
447:               hosts = Helpers.extract_hosts_from_array(flags[:nodes])
448:             elsif flags[:json]
449:               hosts = Helpers.extract_hosts_from_json(flags[:json])
450:             end
451: 
452:             raise "Could not find any hosts in discovery data provided" if hosts.empty?
453: 
454:             @discovered_agents = hosts
455:             @force_direct_request = true
456: 
457:           # if an identity filter is supplied and it is all strings no regex we can use that
458:           # as discovery data, technically the identity filter is then redundant if we are
459:           # in direct addressing mode and we could empty it out but this use case should
460:           # only really be for a few -I's on the CLI
461:           #
462:           # For safety we leave the filter in place for now, that way we can support this
463:           # enhancement also in broadcast mode
464:           elsif options[:filter]["identity"].size > 0
465:             regex_filters = options[:filter]["identity"].select{|i| i.match("^\/")}.size
466: 
467:             if regex_filters == 0
468:               @discovered_agents = options[:filter]["identity"].clone
469:               @force_direct_request = true if Config.instance.direct_addressing
470:             end
471:           end
472:         end
473: 
474:         # All else fails we do it the hard way using a traditional broadcast
475:         unless @discovered_agents
476:           @stats.time_discovery :start
477: 
478:           # if compound filters are used the only real option is to use the mc
479:           # discovery plugin since its the only capable of using data queries etc
480:           # and we do not want to degrade that experience just to allow compounds
481:           # on other discovery plugins the UX would be too bad raising complex sets
482:           # of errors etc.
483:           @client.discoverer.force_discovery_method_by_filter(options[:filter])
484: 
485:           actual_timeout = options[:disctimeout] + @client.timeout_for_compound_filter(options[:filter]["compound"])
486:           if actual_timeout > 0
487:             @stderr.print("Discovering hosts using the %s method for %d second(s) .... " % [@client.discoverer.discovery_method, actual_timeout]) if verbose
488:           else
489:             @stderr.print("Discovering hosts using the %s method .... " % [@client.discoverer.discovery_method]) if verbose
490:           end
491: 
492:           @client.options = options
493: 
494:           # if the requested limit is a pure number and not a percent
495:           # and if we're configured to use the first found hosts as the
496:           # limit method then pass in the limit thus minimizing the amount
497:           # of work we do in the discover phase and speeding it up significantly
498:           if @limit_method == :first and @limit_targets.is_a?(Fixnum)
499:             @discovered_agents = @client.discover(@filter, options[:disctimeout], @limit_targets)
500:           else
501:             @discovered_agents = @client.discover(@filter, options[:disctimeout])
502:           end
503: 
504:           @stderr.puts(@discovered_agents.size) if verbose
505: 
506:           @force_direct_request = @client.discoverer.force_direct_mode?
507: 
508:           @stats.time_discovery :end
509:         end
510: 
511:         @stats.discovered_agents(@discovered_agents)
512:         RPC.discovered(@discovered_agents)
513: 
514:         @discovered_agents
515:       end

[Source]

     # File lib/mcollective/rpc/client.rb, line 332
332:       def discovery_method=(method)
333:         @discovery_method = method
334: 
335:         if @initial_options[:discovery_options]
336:           @discovery_options = @initial_options[:discovery_options]
337:         else
338:           @discovery_options.clear
339:         end
340: 
341:         @client.options = options
342:         @discovery_timeout = discovery_timeout
343:         reset
344:       end

[Source]

     # File lib/mcollective/rpc/client.rb, line 346
346:       def discovery_options=(options)
347:         @discovery_options = [options].flatten
348:         reset
349:       end

[Source]

     # File lib/mcollective/rpc/client.rb, line 327
327:       def discovery_timeout
328:         return @initial_options[:disctimeout] if @initial_options[:disctimeout]
329:         return @client.discoverer.ddl.meta[:timeout]
330:       end

Sets the fact filter

[Source]

     # File lib/mcollective/rpc/client.rb, line 359
359:       def fact_filter(fact, value=nil, operator="=")
360:         return if fact.nil?
361:         return if fact == false
362: 
363:         if value.nil?
364:           parsed = Util.parse_fact_string(fact)
365:           @filter["fact"] << parsed unless parsed == false
366:         else
367:           parsed = Util.parse_fact_string("#{fact}#{operator}#{value}")
368:           @filter["fact"] << parsed unless parsed == false
369:         end
370: 
371:         @filter["fact"].compact!
372:         reset
373:       end

Returns help for an agent if a DDL was found

[Source]

     # File lib/mcollective/rpc/client.rb, line 117
117:       def help(template)
118:         if @ddl
119:           @ddl.help(template)
120:         else
121:           return "Can't find DDL for agent '#{@agent}'"
122:         end
123:       end

Sets the identity filter

[Source]

     # File lib/mcollective/rpc/client.rb, line 383
383:       def identity_filter(identity)
384:         @filter["identity"] << identity
385:         @filter["identity"].compact!
386:         reset
387:       end

Sets and sanity check the limit_method variable used to determine how to limit targets if limit_targets is set

[Source]

     # File lib/mcollective/rpc/client.rb, line 559
559:       def limit_method=(method)
560:         method = method.to_sym unless method.is_a?(Symbol)
561: 
562:         raise "Unknown limit method #{method} must be :random or :first" unless [:random, :first].include?(method)
563: 
564:         @limit_method = method
565:       end

Sets and sanity checks the limit_targets variable used to restrict how many nodes we‘ll target

[Source]

     # File lib/mcollective/rpc/client.rb, line 543
543:       def limit_targets=(limit)
544:         if limit.is_a?(String)
545:           raise "Invalid limit specified: #{limit} valid limits are /^\d+%*$/" unless limit =~ /^\d+%*$/
546: 
547:           begin
548:             @limit_targets = Integer(limit)
549:           rescue
550:             @limit_targets = limit
551:           end
552:         else
553:           @limit_targets = Integer(limit)
554:         end
555:       end

Magic handler to invoke remote methods

Once the stub is created using the constructor or the RPC#rpcclient helper you can call remote actions easily:

  ret = rpc.echo(:msg => "hello world")

This will call the ‘echo’ action of the ‘rpctest’ agent and return the result as an array, the array will be a simplified result set from the usual full MCollective::Client#req with additional error codes and error text:

{

  :sender => "remote.box.com",
  :statuscode => 0,
  :statusmsg => "OK",
  :data => "hello world"

}

If :statuscode is 0 then everything went find, if it‘s 1 then you supplied the correct arguments etc but the request could not be completed, you‘ll find a human parsable reason in :statusmsg then.

Codes 2 to 5 maps directly to UnknownRPCAction, MissingRPCData, InvalidRPCData and UnknownRPCError see below for a description of those, in each case :statusmsg would be the reason for failure.

To get access to the full result of the MCollective::Client#req calls you can pass in a block:

  rpc.echo(:msg => "hello world") do |resp|
     pp resp
  end

In this case resp will the result from MCollective::Client#req. Instead of returning simple text and codes as above you‘ll also need to handle the following exceptions:

UnknownRPCAction - There is no matching action on the agent MissingRPCData - You did not supply all the needed parameters for the action InvalidRPCData - The data you did supply did not pass validation UnknownRPCError - Some other error prevented the agent from running

During calls a progress indicator will be shown of how many results we‘ve received against how many nodes were discovered, you can disable this by setting progress to false:

  rpc.progress = false

This supports a 2nd mode where it will send the SimpleRPC request and never handle the responses. It‘s a bit like UDP, it sends the request with the filter attached and you only get back the requestid, you have no indication about results.

You can invoke this using:

  puts rpc.echo(:process_results => false)

This will output just the request id.

Batched processing is supported:

  printrpc rpc.ping(:batch_size => 5)

This will do everything exactly as normal but communicate to only 5 agents at a time

[Source]

     # File lib/mcollective/rpc/client.rb, line 214
214:       def method_missing(method_name, *args, &block)
215:         # set args to an empty hash if nothings given
216:         args = args[0]
217:         args = {} if args.nil?
218: 
219:         action = method_name.to_s
220: 
221:         @stats.reset
222: 
223:         @ddl.validate_rpc_request(action, args) if @ddl
224: 
225:         # if a global batch size is set just use that else set it
226:         # in the case that it was passed as an argument
227:         batch_mode = args.include?(:batch_size) || @batch_mode
228:         batch_size = args.delete(:batch_size) || @batch_size
229:         batch_sleep_time = args.delete(:batch_sleep_time) || @batch_sleep_time
230: 
231:         # if we were given a batch_size argument thats 0 and batch_mode was
232:         # determined to be on via global options etc this will allow a batch_size
233:         # of 0 to disable or batch_mode for this call only
234:         batch_mode = (batch_mode && Integer(batch_size) > 0)
235: 
236:         # Handle single target requests by doing discovery and picking
237:         # a random node.  Then do a custom request specifying a filter
238:         # that will only match the one node.
239:         if @limit_targets
240:           target_nodes = pick_nodes_from_discovered(@limit_targets)
241:           Log.debug("Picked #{target_nodes.join(',')} as limited target(s)")
242: 
243:           custom_request(action, args, target_nodes, {"identity" => /^(#{target_nodes.join('|')})$/}, &block)
244:         elsif batch_mode
245:           call_agent_batched(action, args, options, batch_size, batch_sleep_time, &block)
246:         else
247:           call_agent(action, args, options, :auto, &block)
248:         end
249:       end

Creates a suitable request hash for the SimpleRPC agent.

You‘d use this if you ever wanted to take care of sending requests on your own - perhaps via Client#sendreq if you didn‘t care for responses.

In that case you can just do:

  msg = your_rpc.new_request("some_action", :foo => :bar)
  filter = your_rpc.filter

  your_rpc.client.sendreq(msg, msg[:agent], filter)

This will send a SimpleRPC request to the action some_action with arguments :foo = :bar, it will return immediately and you will have no indication at all if the request was receieved or not

Clearly the use of this technique should be limited and done only if your code requires such a thing

[Source]

     # File lib/mcollective/rpc/client.rb, line 144
144:       def new_request(action, data)
145:         callerid = PluginManager["security_plugin"].callerid
146: 
147:         raise 'callerid received from security plugin is not valid' unless PluginManager["security_plugin"].valid_callerid?(callerid)
148: 
149:         {:agent  => @agent,
150:          :action => action,
151:          :caller => callerid,
152:          :data   => data}
153:       end

Provides a normal options hash like you would get from Optionparser

[Source]

     # File lib/mcollective/rpc/client.rb, line 519
519:       def options
520:         {:disctimeout => @discovery_timeout,
521:          :timeout => @timeout,
522:          :verbose => @verbose,
523:          :filter => @filter,
524:          :collective => @collective,
525:          :output_format => @output_format,
526:          :ttl => @ttl,
527:          :discovery_method => @discovery_method,
528:          :discovery_options => @discovery_options,
529:          :config => @config}
530:       end

Resets various internal parts of the class, most importantly it clears out the cached discovery

[Source]

     # File lib/mcollective/rpc/client.rb, line 397
397:       def reset
398:         @discovered_agents = nil
399:       end

Reet the filter to an empty one

[Source]

     # File lib/mcollective/rpc/client.rb, line 402
402:       def reset_filter
403:         @filter = Util.empty_filter
404:         agent_filter @agent
405:       end

[Validate]